基准优化不等于通用编码能力：需多样化评估
原标题：Don't Claim Benchmark-Oriented Optimization Improves General Coding Capability -- Diverse Evaluation Is Required
AI 摘要
该论文指出，将SWE-bench等少数编码基准的分数视为通用编码能力的证据存在意义差距，因为针对基准的优化可能只提升任务特定性能。作者通过Django案例研究基准评估了后训练模型，发现SWE-bench优化带来的提升无法泛化到其他编码任务，甚至在同一仓库内也如此。论文呼吁采用多样化评估方法，并建立能力分类法和持续基准维护，以避免误导性的研究结论和部署决策。
正文节选
Don’t Claim Benchmark-Oriented Optimization Improves General Coding Capability — Diverse Evaluation Is Required Abstract Post-training papers, model cards, and blog posts often treat scores on a small set of coding benchmarks (e.g., SWE-bench and LiveCodeBench) as evidence of broad “coding capability”, both for research artifacts and user-facing systems. We argue that optimization for these benchmarks leads to measuring task-specific performance, creating a meaning gap between measured scores an