先处理再检索：探究视觉语言模型模态差距的起源
原标题：Process then Retrieve: Investigating the Origins of the Modality Gap in Vision-Language Models
AI 摘要
该论文研究视觉语言模型（VLM）中模态差距的成因，发现嵌入层文本token的语义密度远低于视觉token，导致模型早期层优先处理文本表征、将视觉信息整合推迟到中后层。作者通过机制分析，将视觉token位置替换为语义最近的文本token后，早期层对这些位置的注意力增强，表明初始语义密度差异影响跨模态注意力分配。研究为缓解模态差距、改进VLM多模态推理提供了机制层面的依据。
正文节选
Process then Retrieve: Investigating the Origins of the Modality Gap in VLMs Abstract. Understanding the origins of the modality gap in vision–language models remains a significant challenge. This work investigates whether differences in information density between modality-specific tokens at the embedding layer contribute to this imbalance. We show that text tokens enter the model with much lower semantic density than visual tokens, driving early layers to focus on processing text representatio