预训练语言模型循环深度改造：安装、外推、迁移与保留
原标题：Retrofitting Recurrent Depth into a Pretrained Language Model: Installation, Extrapolation, Transfer, and Retention at Two Parameter Budgets
AI 摘要
该研究将预训练语言模型 Qwen2.5-0.5B-Instruct 改造为循环深度架构，通过权重共享的循环块和可训练桥接层实现迭代潜在计算。在两种参数预算（6M 适配器和 180M 全块）下，模型在 ARC 基准上表现不逊于基线，并能外推至监督深度的 1.5 倍。与同规模自回归模型相比，循环模型在深度推理上准确率更高（84% vs 72%），速度更快，且保留能力更强。研究还发现逆任务训练会导致灾难性干扰，深度选择仍是开放问题。
正文节选
Retrofitting Recurrent Depth into a Pretrained Language Model: Installation, Extrapolation, Transfer, and Retention at Two Parameter Budgets Abstract A dense, pretrained language model can be retrofitted with recurrent depth and learn an iterative latent transition that persists after outcome-only annealing. Qwen2.5-0.5B-Instruct is split into a Prelude, a weight-tied Recurrent Block, and a Coda, with an identity-preserving one-loop path and a trainable bridge that re-injects the Prelude represe