基准饱和后利用解答记录评估模型能力

AI 摘要

该研究探讨基准测试饱和后如何继续评估模型能力。作者利用历史基准和模型发布数据回溯模拟饱和，测试仅访问已饱和基准中正确解答记录的方法能否预测模型在未饱和任务上的能力。结果显示，饱和基准的解答记录仍保留超出准确率的能力信息，其中成对LLM评判效果最佳，预测误差比仅用饱和基准分数的基线降低5.27；但现有协议在分布偏移下并不总是稳健。

正文节选

Evaluating Model Capabilities After Benchmark Saturation Abstract Rapid advances in AI capabilities are causing benchmarks to saturate. Once several models approach perfect scores, a benchmark loses its power to differentiate between them, and with it its utility for informing AI safety cases and frontier policies. Yet benchmark items produce far more information than the scores we typically extract from them: a full transcript of how each model solved the task. We investigate whether these tran

发布时间：2026-10-17 12:00

抓取时间：2026-10-16 12:10

来源机构：arXiv