代码LLM离线后训练：性能、效率与崩溃
原标题：Performance, Efficiency and Collapse -- Advantages and Challenges in Offline Post-training of Code LLMs
AI 摘要
该研究探讨代码大语言模型能否完全离线进行强化学习后训练，而不依赖在线采样。作者在Qwen Coder、DeepSeek Coder和CodeLLaMa等0.5B至7B参数模型上，使用CodeNet数据集进行实验，发现仅需数小时训练即可显著提升零样本代码生成性能。研究还发现离线RL对学习率和训练轮数高度敏感，长时间训练会导致模型崩溃，其主因是logit方差而非在线RL中的优势函数方差，并提出基于logit方差的早停诊断方法。
正文节选
Performance, Efficiency and Collapse – Advantages and Challenges in Offline Post-training of Code LLMs Abstract Post-training with reinforcement learning (RL) is a critical phase in the development of code-generating large language models (LLMs), as it ensures adherence to instructions and the production of functionally correct code. This process typically requires computationally intensive code sample generation from Transformer-based LLMs and substantial GPU-CPU communication for sequence veri