CARAT：材料大模型是在推理还是背诵？
原标题：CARAT: Do Materials LLMs Reason or Recite?
AI 摘要
该论文提出 CARAT 基准，用于检验材料领域大语言模型在回答晶体结构问题时究竟是在进行结构推理，还是仅仅复述输入中已印出的答案（readback）。CARAT 将同一问题渲染为八种匹配视图，并加入配对评估、反捷径测试和统计规则，在 562 种材料、1024 道匹配问题上测试 27B 模型。结果显示 GraphSpace 视图相对公式输入有 17.3 个百分点优势，但该优势主要来自基线视图缺失必要信息，而非证据呈现方式；模型在 95.6% 的配对案例中引用关系却不使用它，经匹配监督后可达 99.8%，删除链接则降至 23.4%。
正文节选
CARAT: Do Materials LLMs Reason or Recite? Abstract When a materials LLM answers a question about crystal structure, does it reason from the structure or copy an answer already printed in its input? Accuracy cannot tell: a structural description often prints the very field it is scored against. CARAT holds question and gold answer fixed across eight matched views, names each structural relation separately in GraphSpace, and adds matched fine-tuning, answer masking, evidence injection, paired inf