通过自蒸馏增强基于评分标准的强化学习
原标题：Enhancing Rubric-based RL via Self-Distillation
AI 摘要
Hugging Face 每日论文介绍了一篇关于基于评分标准的强化学习（Rubric-based RL）的研究。论文指出该方法的两个失败模式：未探索标准（UC）和受抑制标准（SC），其中SC在超过57%的训练样本中出现。作者提出Criterion-Distilled Policy Optimization（CriPO）方法，通过策略内自蒸馏和令牌级优势修正同时解决UC和SC，避免训练-推理不匹配。在医学和科学基准测试中，CriPO优于现有方法，且优化步骤减少约2倍。
正文节选
Enhancing Rubric-based RL via Self-Distillation Abstract Rubric-based RL has recently shown promise in improving LLMs on open-ended tasks. A widely recognized limitation of rubric-based RL is limited exploration: criteria that no rollout manages to satisfy (Unexplored Criteria, UC) receive no optimization signal. Recent methods address this by incorporating rubric information as external guidance during rollout, yet they introduce a train-inference mismatch: the policy is optimized on rollouts p