一次权重更新买到了什么：跨训练阶段的杠杆率
原标题：What a Weight Update Buys: Leverage Across Training Stages
AI 摘要
该论文提出「杠杆率」（leverage）指标，用Fisher二次型精确衡量单位权重更新带来的行为变化，并以同尺寸随机方向为基准归一化。作者在Olmo-3、Tülu-3、Zephyr、OLMo-2等已发布检查点上测得杠杆率跨越四个数量级：预训练接近随机水平，SFT约5倍，RL-Zero/RLVR为400–1500倍，DPO为400–23000倍。研究还发现预训练更新累积为噪声，bf16无fp32主副本训练会冻结36–99%坐标并最多降低七倍杠杆率，且杠杆率可预测DARE合并的相对损伤。
正文节选
What a Weight Update Buys: Leverage Across Training Stages Abstract Post-training moves a language model’s weights by about a thousandth of their norm and changes its behavior sharply; pretraining moves them by a third and buys a slow drift. We introduce leverage, an exact measure of what a unit of weight change buys: the Fisher quadratic form of an update, computed with one forward-mode pass per direction, divided by the same form for random directions of the same size at the same checkpoint. L