外部引导何时助益LLM推理？引导增强GRPO的偏差-方差理论
原标题：When Does External Guidance Help LLM Reasoning? A Bias-Variance Theory of Guidance-Augmented GRPO
AI 摘要
该论文提出 Guidance-Augmented GRPO (GA-GRPO) 统一理论框架，将外部引导建模为改写问题分布的随机引导算子，把由此产生的策略梯度估计量分析为有偏的 on-policy 估计量，其偏差由引导分布与策略自身分布之间的全变差散度界定。框架涵盖 vanilla GRPO、LUFFY、ExPO、PAPO、TAPO 等作为特例，并证明收敛速率 O(1/sqrt(T))、推导出 MSE 最优引导权重闭式解，以及匹配的 minimax 下界。在 Qwen2.5-Math-7B-Base 上九个数学与 OOD 基准实验中，最优权重 GA-GRPO 匹配或超越 TAPO、LUFFY、ExPO 与 vanilla GRPO，同时节省 31% GPU 小时。
正文节选
When Does External Guidance Help LLM Reasoning? A Bias-Variance Theory of Guidance-Augmented GRPO Abstract Reinforcement learning with verifiable rewards (RLVR) has become the dominant paradigm for eliciting multi-step reasoning in large language models, and a recent wave of methods (LUFFY, ExPO, PAPO, TAPO) further augments RL with external guidance—expert traces, self-explanations, or retrieved thought patterns. Although each method reports empirical gains, none provides convergence rates, bi