AuditKIT：面向大模型与智能体系统的统一评测框架
原标题：AuditKIT: A Unified Evaluation Spine for Benchmarking, Judging, and Comparing Large Language Models and Agentic Systems
AI 摘要
论文提出 AuditKIT，一个面向大语言模型与智能体系统的开源评测库，统一了基准评测、模型评判、性能测量、工具使用评估和 RAG 评测接口。它要求每个指标声明 MAXIMIZE 或 MINIMIZE 方向并在比较中强制执行，用 SHA-256 指纹作为缓存与比较键，并将无法获取的证据标记为未评分而非计为失败。作者通过七个案例研究说明答案抽取、服务行为、检索指标和轨迹结构如何改变评测结果的解读，但明确表示这些案例不用于确立模型排名。
正文节选
1 Introduction Evaluating a language model involves more than a score on one benchmark. Different applications call for task accuracy, generation quality, runtime efficiency, or safety, and each uses its own measurements. Agents add further requirements [48, 38, 20]. Their behavior depends on tool selection, argument correctness, execution order, retrieved contexts, and changes to the external environment, not only on generated text. A model can write a plausible final answer while the underlyin