RubricCraft：从解法分歧中归纳评分标准
原标题：RubricCraft: Where Solutions Diverge, Criteria Emerge
AI 摘要
论文提出 RubricCraft，一种基于候选解差异自动构建文本评分标准（rubric）的方法，用于评估产出型语言智能体。它通过比较多个独立求解轨迹的差异来生成、修复并重新验证评分标准，从而发现不公平、不完整、无区分度等缺陷。在 SpreadsheetBench v2 上，相比同一模型仅凭提示词编写的 rubric，RubricCraft 的错误输出通过率更低（29.5% 对 42.6%），且不会误判基准检查器认可的正确答案。
正文节选
RubricCraft: Where Solutions Diverge, Criteria Emerge Abstract Evaluating language agents on open-ended, artifact-producing tasks is difficult: model-based graders can be inconsistent, while exhaustive programmatic checks are often impractical. Textual rubrics make evaluation criteria explicit, but writing them requires domain expertise. Rubrics generated from task descriptions alone can miss requirements and failure modes that become apparent in attempted solutions. We introduce RubricCraft, wh