Li2SAO：组相对演员-评论家自举提升轻量单轨迹异步RL优化
原标题：Li$^{2}$SAO: Lifting Light Singl-Rollout Asynchronous RL Optimization via Group-Relative Actor-Critic Bootstrapping
AI 摘要
该论文针对异步强化学习（RL）中单轨迹优化（SAO）依赖闭源预训练价值模型的问题，提出轻量复现版 LiSAO，并分析发现 GRPO 在单轮推理任务上更优、LiSAO 在多轮智能体任务上更优，但 LiSAO 因 critic 冷启动在训练早期明显落后。为此作者提出 Li2SAO，在 critic 冷启动阶段用组相对优势训练 actor、用组中心回报训练 critic，待 critic 校准后切换为单轨迹 GAE。实验表明 Li2SAO 在训练早期追平或超越 GRPO，并进一步提升后期性能上限，代码已开源。
正文节选
Li2SAO: Lifting Light Single-Rollout Asynchronous RL Optimization via Group-Relative Actor-Critic Bootstrapping Abstract Asynchronous reinforcement learning (RL) is becoming a mainstream paradigm for post-training large language models (LLMs), as it improves training efficiency by updating the model as rollouts arrive. Recent work argues that group-wise sampling in GRPO does not naturally fit asynchronous RL training because rollout lengths follow a long-tailed distribution, and proposes Single-