小语言模型能否知道自己不知道？语义熵作为置信信号
原标题：Do small language models know what they don't know?
AI 摘要
该研究探讨了参数量低于30亿的小语言模型（SLM）能否通过熵信号判断自身不确定性。作者在7个模型对和5个NLU基准上评估了7种方法，发现91%的数据集-模型组合中token级熵几乎为零，无法作为置信信号；而语义熵（多次采样并按语义聚类）能恢复有效的不确定性信号。利用语义熵将不确定查询路由到更大的专家模型，准确率最高提升50个百分点，且跨家族路由（如SmolLM 360M到Phi-3.5-mini）平均提升22.0%，优于同家族路由的6.8%。
正文节选
Can Small Language Models Know What They Don’t Know? Semantic Entropy as a Confidence Signal for Sub-3B Parameter Models Abstract We explore whether entropy-based confidence signals can be leveraged to improve the accuracy of Small Language Models (SLMs) with fewer than 3 billion parameters, running entirely on consumer hardware. We evaluate seven distinct approaches, including token-level entropy early stopping, semantic entropy estimation, and uncertainty-aware routing to larger expert models,