TestJack：通过评估器演化审计编码基准测试结果
原标题：TestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator Evolution
AI 摘要
哥伦比亚大学、UC Berkeley 和乔治梅森大学的研究者提出 TestJack，一个通过演化评估器来审计编码基准测试结果的自动化框架。它针对每个通过固定单元测试的试验，依据任务提示生成新测试，并用真实补丁验证测试有效性，从而发现被误判为正确的试验。在 6 个前沿模型后端和 5 个基准（如 DeepSWE、SWE Marathon）上，约 34.4% 被判定正确的试验实际违反任务要求，整体解决率从 50.6% 降至 33.2%。
正文节选
promptbox TestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator Evolution Abstract Large language model (LLM) agents are rapidly reshaping software engineering, accompanied by an explosion of new code benchmarks. Yet nearly all existing benchmarks still rely on the same decades-old criterion: a solution is correct if it passes a fixed set of unit tests. Such tests are often insufficient: they check only part of what the task requires, so ag