音频大模型多模态语音情绪识别的不一致性感知评估
原标题：Incongruence-Aware Evaluation of Multimodal Speech Emotion Recognition in Audio Large Language Models
AI 摘要
该研究提出一种基于高斯混合模型（GMM）的跨模态不一致性评分方法，将文本与音频映射到共享的效价-唤醒-支配（VAD）空间，把两者距离转化为连续的不一致性概率，并与人类判断进行验证（Spearman r=0.65）。作者在eNTERFACE'05、MELD和CMU-MOSEI三个语料上评估三个音频大模型，发现在控制真实情绪和特征空间距离后，只要音频存在，不一致性就会提高模型出错几率；而用相同特征训练的监督分类器表现相反。研究还发现纯文本效应不可靠，仅更换读取转录文本的模型就会使效应符号反转，表明困难源于模型融合原始证据的方式而非信号本身。
正文节选
Incongruence-Aware Evaluation of Multimodal Speech Emotion Recognition in Audio Large Language Models Abstract Voice-interactive large language models process both what a speaker says and how they say it, yet the two channels do not always agree. We ask whether that disagreement degrades speech emotion recognition. Mapping text and audio into a shared valence-arousal-dominance space, we convert the distance between them into a continuous, threshold-free incongruence score with a two-component Ga