SCI:测量赋予信念是否组织语言模型认知
原标题:The Schema Coherence Inventory: Measuring Whether an Assigned Belief Organizes a Language Model's Cognition
AI 摘要
研究者提出 Schema Coherence Inventory(SCI),用 96 道强制选择题测量被赋予的信念是否真正组织语言模型的行为,而非仅带来情绪基调。该工具在 5 个 1.7B–14B 参数骨干模型、107 个条件下验证,信念适配器在每个骨干上均表现出目标特异性(+17.7 分,30/30 信念单元为正),临床训练评分者恢复关键信念的准确率为 79.5%(随机为 16.7%)。应用于 6 个已发表的病人模拟器时,没有一个比一句核心信念陈述更好地承载其设定信念。作者公开了题目、评分代码和简版。
正文节选
The Schema Coherence Inventory: Measuring Whether an Assigned Belief Organizes a Language Model’s Cognition Abstract Persona-conditioned language models are used as simulated patients and affective agents, but there is no validated way to ask whether a belief assigned to such a model organizes its behaviour or whether the model has merely adopted a mood. We present the Schema Coherence Inventory (SCI): 96 forced-choice items at eight levels of Beck’s cognitive model, each offering six replies ke