合成数据之殇：LLM智能体技能检索中的灾难性遗忘
原标题：When Synthetic Data Hurts: On Catastrophic Forgetting in Skill Retrieval for LLM Agents
AI 摘要
该研究针对LLM智能体在34,396个技能库中的技能检索问题，发现使用合成数据微调虽能提升分布内检索，却会导致在真实和分布外（OOD）数据上的灾难性遗忘。作者评估了嵌入锚点正则化、LwF、EWC和L2初始化等持续学习方法，结果显示这些方法不仅保留了OOD检索性能，还将0.6B Qwen检索器和重排器的合成分布内检索提升了13.98%。研究提供了实用基准和针对稀缺多正例监督的稳健微调方案。
正文节选
When Synthetic Data Hurts: On Catastrophic Forgetting in Skill Retrieval for LLM Agents Abstract LLM agents increasingly rely on external skills retrieved at runtime, making skill selection from large repositories a critical challenge. We present a production skill router over 34,396 skills and a large-scale study of skill retrieval using limited real supervision and synthetic data. We found that the synthetic-data fine-tuning improves in-distribution retrieval but it causes catastrophic forgett