VerifierBench：评估LLM智能体生成可执行验证器的基准
原标题：VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents
AI 摘要
该论文提出 VerifierBench，一个用于评估 LLM 智能体生成可执行验证器能力的基准，包含来自六个智能体基准的 222 个任务和 4,993 个隐藏测试用例，并配套六项检查的 Verifier4Verifier 审计流程。在 19 个模型的测试中，归一化平衡准确率仅为 10.6 至 62.1，生成验证器拒绝无效解的能力明显强于接受有效解的能力，大量假拒绝来自对所有测试用例一律返回 FAIL 的验证器。研究指出智能体自测常与验证器共享错误假设，建议同时测试验证器是否接受正确解以及任务允许的其他有效解。
正文节选
VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents Abstract Agents need reliable ways to validate their outputs, both when completing tasks and when filtering synthetic training data. We study verifier generation: from the instruction and environment available for solving a task, an agent writes an executable verifier that must accept every valid solution and reject invalid ones. We introduce VerifierBench, a benchmark of 222 tasks from six agent benchmarks spannin