参数空间探索通过变分学习改进 LLM 强化学习
原标题：Parameter Exploration for RLVR via Variational Learning
AI 摘要
Hugging Face 每日论文介绍了一篇关于参数空间探索的研究，提出了一种名为 Perturbed Parameter Policy Optimization (3PO) 的方法族，通过从后验分布中采样不同策略来生成轨迹，以改进 LLM 的强化学习。实验表明，在 OLMo-3-1025-7B 和 Qwen2.5-Math-7B 上，3PO 在数学推理和代码生成任务中相比标准 GRPO 持续提升平均下游性能，且计算成本相近，同时减少了训练中的零优势组和错误轨迹。
正文节选
Parameter Exploration for RLVR via Variational Learning Abstract Parameter-space exploration via perturbed policy sampling improves LLM reinforcement learning by diversifying rollouts and reducing training failures compared to action-space methods. Exploration has been a focus of reinforcement learning research for a long time. Recently, there has been growing evidence that it is also an important ingredient in LLM reinforcement learning recipes that can significantly impact downstream performan