审计并修复生产 text-to-SQL 流水线中的 LLM 评判失败
原标题：Auditing and Repairing LLM-as-Judge Failures in a Production Text-to-SQL Pipeline
AI 摘要
该论文审计了一个生产环境 text-to-SQL 流水线中的 LLM-as-judge，发现部署的 gpt-4o-mini 评判器与人工标注一致性很低，在富集样本上过度标记了 77.1% 的人类可信案例，主要失败机制被命名为 Grade-Hallucination。作者测试了替代方案，发现自托管 Qwen3.6-27B 与 Claude Opus 4.7 表现相当但成本约为 1/300，并指出弱评判器与强评判器配对会降低一致性，而三个强评判器一致路由可达到 89.7% 自动覆盖率。该审计方法在 BIRD-financial 上标记了 25.5% 的专家编写 gold SQL 为潜在问题。
正文节选
Auditing and Repairing LLM-as-Judge Failures in a Production Text-to-SQL Pipeline Abstract Production text-to-SQL pipelines often end with an LLM-as-judge whose agreement with human annotators has never actually been measured. When we checked ours, the deployed gpt-4o-mini judge agreed with two-author gold at only Cohen’s on a disagreement-enriched set and on a uniform-random spot-check, over-flagging 77.1% of the human-faithful cases in the enriched set. Most of its over-flags trace back to a s