教师是方向而非终点：在线策略蒸馏中RL表示残差的外推
原标题：Paper page - The Teacher Is a Direction, Not a Destination: Extrapolating RL-Induced Representation Residuals in On-Policy Distillation
AI 摘要
该论文提出 RIDE（RL-Induced Direction Extrapolation）方法，用于在线策略蒸馏（OPD）。作者观察到强化学习会相对基础检查点改变模型内部表示，且该变化方向可在每一层测量，因此 RIDE 在表示空间中直接外推 RL 引起的残差：在每层和每个 token 位置计算教师模型与其 RL 前检查点的残差，并将学生隐藏状态回归到沿该残差超越教师的目标上。在四组不同规模、架构和预训练来源的基础/RL 教师对上，RIDE 均接近或超过 RL 训练教师，且优于输出空间外推方法。
正文节选
The Teacher Is a Direction, Not a Destination: Extrapolating RL-Induced Representation Residuals in On-Policy Distillation Abstract On-policy distillation (OPD) trains a student to match the teacher's next-token distributions on the student's own trajectories and has yielded substantial empirical gains. Generalized variants allow the student to surpass the teacher by extrapolating an implicit reward in output space. The language-model head, however, attenuates this change anisotropically: much o