Proxy Confidence：用代理模型对数概率审计黑盒 LLM Agent
原标题：Proxy Confidence: Auditing Black-Box LLM Agents with a Surrogate's Log-Probabilities
AI 摘要
该论文提出一种名为 Proxy Confidence 的黑盒 LLM Agent 审计方法：由于前沿聊天 API 隐藏 token 概率且模型近乎确定性导致自一致性失效，作者用一个低成本开源权重代理模型并行读取相同上下文、schema 和候选动作，通过 teacher forcing、request-PMI、判别式判定和工具选择竞争等互补读出方式，从代理模型的 log-probabilities 中恢复置信信号。该信号无需训练、无需访问 Agent 内部，仅需一次 prefill，在困难编码任务上 AUROC 显著优于 Agent 自述置信度，并可用于实时门控人工审核和置信反馈提升端到端任务成功率。
正文节选
Interpreting Agent Behavior (IAB) Proxy Confidence: Auditing Black-Box LLM Agents with a Surrogate’s Log-Probabilities Abstract A deployed LLM agent emits tool calls, queries, and code that can be silently wrong—and by the time the error surfaces, the action has already run. The frontier chat APIs these agents run on offer no reliable way to tell which calls to trust. They hide the model’s own token probabilities. The agent’s stated confidence barely beats chance on the mistakes that matter. And