ELR：消除单循环税的弹性潜在循环语言模型
原标题：One Model to Loop Them All: Elastic Latent Recurrence for Language Models
AI 摘要
该论文提出 Elastic Latent Recurrence（ELR）训练方法，解决循环语言模型在单次循环时性能低于同规模非循环模型的「单循环税」问题。ELR 通过门控状态更新、优化器更新内混合循环次数、以及按循环索引和预算进行条件化三项设计，使单循环时性能与基线相当，增加循环次数则持续提升精度。在 4B 稠密和 30B-A3B MoE 混合模型上，四循环分别带来 7.25 和 4.75 分提升，并产出 Koi-4B 检查点。
正文节选
One Model to Loop Them All: Elastic Latent Recurrence for Language Models Abstract Abstract A latent recurrent language model applies a shared block of layers repeatedly, so one checkpoint can trade inference compute for accuracy by changing the loop count. At one loop, however, such models typically fall well short of a non-recurrent model of the same size, a one-loop tax that makes a separate non-recurrent model the better choice whenever latency matters. We present Elastic Latent Recurrence (