BenchChecker：评估 LLM 修复 bug 基准的可信度
原标题：BenchChecker: Assessing the Credibility of Bug-Fixing Benchmarks for LLMs
AI 摘要
该论文提出 BenchChecker 框架，用于检测 LLM 修复 bug 基准测试中的数据污染问题。它采用两阶段取证检测：先通过代码前缀探测判断仓库是否出现在训练集中，再通过相似度对齐与测试套件验证补丁是否在训练截止快照中持续存在。在 StarCoder-15B、StarCoder2-15B 和 CodeGen2-16B 上评估，AUC 约 0.9；应用于 SWE-bench Verified 和 Multi-SWE-bench 后发现，过滤污染样本会使多数 LLM 在中等难度任务上的解决率相对下降超过 20%。
正文节选
BenchChecker: Assessing the Credibility of Bug-Fixing Benchmarks for LLMs Abstract While Large Language Models (LLMs) show promise for Automatic Program Repair (APR), existing benchmarks suffer from data contamination, in which test samples appear in training sets, artificially inflating performance. To address this, we propose BenchChecker, a framework compatible with both open-source and closed-source models that supports more credible evaluation by detecting such contamination. Grounded in th