现代Transformer是隐式混合体：从功能分化到原则性混合架构设计
原标题：Modern Transformers Are Implicit Hybrids: From Functional Differentiation to Principled Hybrid Architecture Design
AI 摘要
该研究通过干预式指标RFIS和RPD，揭示了基于RoPE的Transformer中头部功能分化为检索型和位置型两类，并识别出全局位置带（GPBand）作为功能边界。基于此提出设计原则，并实例化为头部级混合架构（HwH），采用无位置编码的全注意力进行全局检索、线性注意力进行局部位置建模。实验表明，HwH在保持语言建模和常识推理能力的同时，提升了检索性能，并显著增强了零样本长上下文外推能力。
正文节选
Modern Transformers Are Implicit Hybrids: From Functional Differentiation to Principled Hybrid Architecture Design Abstract Hybrid architectures combining Full Attention (FA) and Linear Attention (LA) are increasingly prominent, yet their allocation remains largely heuristic. We seek an evidence-grounded basis in the head-level functional organization learned by modern RoPE-based Transformers. Behavioral retrieval and local-streaming probes reveal useful tendencies but do not yield a complete ta