代码理解是编码智能体的瓶颈
原标题：Code Understanding is a Bottleneck for Coding Agents
AI 摘要
微软研究院提出 Cabra，一个用于严格评估编码智能体的合成任务框架，通过有向无环调用图从零生成任务，并在函数遍历、搜索、运行时解析和指令遵循四个维度上控制任务规模。研究在 6,840 个任务上测试了 8 个 LLM 和 6 个编码智能体，发现 LLM 准确率随任务规模下降，但智能体通过调用 grep 等工具可保持接近满分；进一步分析 SWE-bench Verified 轨迹表明，智能体用于理解代码的工具调用次数比编辑代码行数更能预测任务难度。作者据此指出代码理解是编码智能体的瓶颈，并呼吁用 Cabra 这类可控合成评估补充 SWE-bench 式基准。
正文节选
inlinecode Code Understanding is a Bottleneck for Coding Agents Abstract Repository benchmarks (e.g., SWE-bench) for coding agents often assume that lines of code edited can predict task difficulty, but such datasets’ poor control over code and task types makes it hard to know which abilities truly drive agent errors. We present Cabra: a Coding Ability Blueprint for Rigorous Agent evaluation. Cabra builds tasks from scratch as call graph transformations and scales difficulty via a task size para