首个token并非判决：不生成直接读取LLM评判的隐藏代价
原标题：The First Token Is Not the Verdict: Hidden Costs of Reading LLM Judges Without Generating
AI 摘要
该论文研究从LLM评判模型首个生成token的logits直接读取判决（即强制读取）的做法，发现这种廉价方法会系统性高估位置偏差，因为评判模型并非总以判决token开头。在Qwen3、Llama-3.1-8B和Phi-3.5-mini等模型及JudgeBench、LLMBar基准上，强制读取在评判模型未主动给出判决时返回的是先展示的响应而非真实判断，导致位置偏差被夸大，而评判准确率几乎不受影响。作者建议在报告位置偏差时同时报告评判模型以判决token开头的比率。
正文节选
Can We Trust the Judge? (JUDGe) @ NeurIPS 2026 The First Token Is Not the Verdict: Hidden Costs of Reading LLM Judges Without Generating Abstract Reading an LLM judge’s verdict from the logits of its first generated token is cheap, requires no generation, and is exactly what constrained decoding and likelihood-scoring evaluation harnesses produce. We show that this readout distorts position bias in one direction: it overstates it in every condition we test, so figures obtained this way behave as