Parcae：稳定循环模型以更少参数实现更高质量
原标题：We present Parcae, one of the first stable architectures for looped language models, achieving the quality of a Transformer twice the size with clean, predictable training. Parcae creates a new medium
AI 摘要
Together AI 发布了 Parcae，一种稳定的循环语言模型架构，通过增加循环次数而非参数规模来提升模型质量。Parcae 在 770M 参数下达到了 1.3B 参数 Transformer 的性能，并建立了首个循环缩放定律。该架构通过参数化注入方式确保稳定性，解决了循环模型训练不稳定的问题，为内存受限的边缘设备训练提供了新路径。
正文节选
We present Parcae, one of the first stable architectures for looped language models, achieving the quality of a Transformer twice the size with clean, predictable training. Parcae creates a new medium to scale quality by increasing recurrence rather than purely scaling data, opening up an efficient frontier for training memory-constrained on-device models. Getting the most out of your parameters Traditional scaling laws tell us that to achieve the best performance, we need to scale FLOPs, often