更多程序还是更多执行？区分LLM Harness中的覆盖与特化
原标题：More Programs or More Rolls? Separating Coverage from Specialization in LLM Harnesses
AI 摘要
该论文研究LLM harness（围绕固定求解器组织提示、工具和控制流的程序）的自动生成，提出一种受控评估方法，区分答案覆盖率、可重复的任务优势与执行前选择带来的增益。在MATH-500上，作者比较八个生成harness与九个字节相同的基线副本，每个执行三次，发现相同程序也能带来oracle余量，生成程序的可重复得分模式主要揭示持续弱点而非稳定优势。冻结选择器仅获得少量百分点增益，且两个群体在相同执行次数下达到oracle覆盖，表明覆盖率和可重复性本身不足以证明有用的任务特化。
正文节选
More Programs or More Rolls? Separating Coverage from Specialization in LLM HarnessesThanks: Code is available at: https://github.com/StatXzy7/harness-eval. Abstract Automated generation of LLM harnesses promises to improve inference through task specialization. Yet additional answer coverage can arise from repeated execution of the same program, making specialization difficult to identify. We introduce a controlled evaluation that separates answer coverage, repeatable task advantages, and gains