Qwen提出GSPO算法：实现语言模型强化学习的可扩展训练
原标题：GSPO: Towards Scalable Reinforcement Learning for Language Models
AI 摘要
Qwen团队提出了一种新的强化学习算法GSPO（Group Sequence Policy Optimization），用于语言模型的规模化训练。GSPO基于序列似然定义重要性比率，并进行序列级裁剪、奖励和优化，相比GRPO具有更高的训练效率、稳定性和对MoE模型的友好性。该算法已成功应用于最新的Qwen3模型（Instruct、Coder、Thinking）的大规模RL训练中，实现了持续的性能提升。GSPO还消除了对Routing Replay策略的依赖，简化了训练基础设施。
正文节选
Introduction Reinforcement Learning (RL) has emerged as a pivotal paradigm for scaling language models and enhancing their deep reasoning and problem-solving capabilities. To scale RL, the foremost prerequisite is maintaining stable and robust training dynamics. However, we observe that existing RL algorithms (such as GRPO) exhibit severe instability issues during long training and lead to irreversible model collapse, hindering further performance improvements with increased compute. To enable s