研究：评测标准左右 LLM 数学辅导成功率测量
原标题：MisconceptDialogue: Evaluation Criteria Shape Measured Success in LLM Math Tutoring
AI 摘要
该论文提出 MisconceptDialogue 全自动评测框架，用 LLM 模拟学生与五个 7B–14B 开源权重 LLM 导师进行 5000+ 轮多轮对话，覆盖检测—诊断—补救全流程。核心发现是评测标准本身对结果影响巨大：基准式精确匹配给出 42.1% 补救成功率，而经人工校准的 LLM-as-judge 给出 81.3%，相差 39.2 个百分点，超过导师模型、教学策略和诊断信息等因素的合计效应。此外，无论告知导师真实误解还是故意替换的误解，得分几乎不变，说明这些模型难以可靠利用所给诊断信息。作者呼吁多标准评分、多评委收敛与人工锚定成为 LLM 辅导评测的标准做法。
正文节选
MisconceptDialogue: Evaluation Criteria Shape Measured Success in LLM Math Tutoring Abstract Large language models are increasingly used as math tutors, raising two questions: can a single LLM detect a student’s error, diagnose the underlying misconception, and remediate it through multi-turn dialogue—and can we trust the numbers that report how well it does? We address both with MisconceptDialogue, a fully automated framework that runs the full pipeline on five open-weight LLM tutors (7B–14B) i