E2E-SWE：评测 LLM 从零构建可运行代码仓库
原标题：E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch
AI 摘要
研究团队发布 E2E-SWE 基准，用于评估 LLM 编码智能体能否仅凭自然语言规格说明和空工作区，从零构建完整可安装的软件仓库。该基准包含覆盖 11 种编程语言的 186 个整仓库生成任务，每个任务由软件工程师与 LLM 协作编写规格与隐藏测试，并经过静态审计和真实模型 rollout 的迭代修复以保证可解性。评测 13 个前沿模型显示 pass@1 从 11.7% 到 67.7%，差异显著且仍有较大提升空间；轨迹分析还揭示出长且前置的推理模式。
正文节选
E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch Abstract Coding agents powered by large language models (LLMs) are evolving from making localized code changes to developing complete software repositories. However, evaluating repository-scale generation remains challenging: tasks must demand system-level reasoning while ensuring that all evaluated behaviors are precisely specified and independent of any particular implementation. We introduce E2E-SWE, a benchmark for evaluat