生产级 LLM 评估：用于安全模型替换的回放管道
原标题：Evaluating LLMs Under Production Parity: A Replay Pipeline for Safe Model Swapping in Conversational Agents
AI 摘要
Hugging Face 博客介绍了一个用于在对话代理中安全替换 LLM 的回放管道，该管道在生产环境下重放已批准的合成交互，仅将模型作为变量。评估了 8 个模型，其中 3 个获批，3 个因幻觉率超标被拒，但逐轮重新计算显示部分拒绝是统计平局或结构性问题。文章强调安全标准应作为独立过滤器，而非加权平均的一部分。
正文节选
Evaluating LLMs Under Production Parity: A Replay Pipeline for Safe Model Swapping in Conversational Agents Public benchmarks don't answer that. They measure the model against generic answer keys, not against your product's behavior, or using your skills, your tools, your tone of voice, and your transactional flows. We built a pipeline to answer this empirically, and the most interesting finding ended up being not about any particular model, but about the design of the verdict. - A replay engine