验证对比而非仅验证评分器：交互式LLM评估中总体准确率可能误导
原标题：Validating the Comparison, Not Just the Scorer: Why Overall Accuracy Can Mislead in Interactive LLM Evaluation
AI 摘要
该论文指出，在交互式LLM评估中，仅用总体准确率验证自动评分器会掩盖不同提示条件下的评分错误差异，从而扭曲提示对比结果。作者在UUP-Bench上用6个模型、180个虚构实体问题和6种提示条件进行审计，发现某对比的表面9.3个百分点提升在两名独立LLM评分者下降至2.3个百分点且置信区间含零。作者建议研究者应验证计划报告的具体差异，而非仅验证评分器整体，并在匹配案例上审计两个提示条件并纳入不确定性。
正文节选
Trustworthy AI for Good (AI4GOOD) Validating the Comparison, Not Just the Scorer: Why Overall Accuracy Can Mislead in Interactive LLM Evaluation Abstract Many LLM studies compare how models respond to different follow-up prompts. For example, one prompt may neutrally ask a model to reconsider, while another insists that an invented film really exists. The difference is used to measure the effect of conversational pressure. An automatic scorer then labels whether each response makes an unsupporte