利用跨模型分歧定位专家投入，加速LLM代码本修订
原标题：Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotation
AI 摘要
该研究提出利用多个LLM在标注同一数据时的分歧来定位代码本中的模糊或缺失规则，从而有针对性地引导专家投入。在数千条辅导会话转录文本上，研究者比较了三种专家反馈方式：代码本验证、问答和理由标注。结果显示，理由标注使LLM标注准确率达到64.9%，优于专家耗时六个月修订的代码本（57.8%），问答方式也达到60.5%。这表明LLM可帮助将代码本修订从数月缩短至数天，同时不牺牲标注性能。
正文节选
Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotation Abstract. Large-scale text annotation brings expert insight to millions of documents, often through a codebook that AI annotators follow. Developing a robust codebook, however, takes months. Large language models (LLMs) could speed this process by applying an early codebook to the data, surfacing cases with strong LLM disagreement, and eliciting expert feed