System-1 决策模型在 LLM Agent Harness 中的配对与自审计评估
原标题：Fast Models, Slow Evidence: A Paired and Self-Audited Evaluation of System-1 Decision Models for LLM Agent Harnesses
AI 摘要
该论文对两种 System-1 决策模型（开源 Laya 与托管 Jev）在 LLM Agent harness 的 11 个决策点上进行了配对评估，使用 18 个公开来源构建的 7,283 个基础案例和 6,640 个鲁棒性变体。结果显示 Jev 在 11 个决策点中的 9 个上显著更准确，但两者在零样本模型路由上均未超过随机水平，在 RAG 相关性门控上持平；Laya 对选项顺序敏感且在同质候选下性能骤降。作者还自我审计了评估流程，发现三处分析错误和一处设计混淆曾扭曲部署结论，例如实际节省仅 4.3% 而非报告的 23.9%。
正文节选
Fast Models, Slow Evidence: A Paired and Self-Audited Evaluation of System-1 Decision Models for LLM Agent Harnesses Abstract Agent harnesses make many small, typed decisions per task: which model to call, which tool to use, whether retrieved text is relevant, whether an input carries an injection. System-1 decision models answer such questions in a single forward pass with class probabilities, promising large cost and latency savings over LLM calls. We present a paired evaluation of an open-wei