多流注意力残差：面向大模型预训练的新残差架构

AI 摘要

该论文提出 Multi-Stream Attention Residuals（MSAR），将 Attention Residuals（AttnRes）沿网络深度轴的单流 softmax 注意力扩展为多头深度注意力。MSAR 通过多尺度因果卷积从序列上下文中获取流多样性，并用轻量自适应混合生成单流层输入，无需持久的多流存储或通信。在 18B 和 28B MoE 规模的 12 个下游基准上，MSAR 一致优于标准残差连接、mHC 和 AttnRes；缩放律实验显示其相对标准残差基线取得 1.58 倍等损失计算加速，高于 AttnRes 的 1.35 倍和 mHC 的 1.30 倍。

正文节选

Abstract Attention Residuals (AttnRes) enables content-dependent retrieval over network depth, which can be viewed as single-stream softmax attention along the depth axis. Inspired by the success of multi-head attention along the sequence axis, we introduce Multi-Stream Attention Residuals (MSAR), an extension of AttnRes that generalizes this mechanism to multi-head depth attention, yielding substantial performance gains with minimal additional overhead. MSAR draws stream diversity from sequence

发布时间：—

抓取时间：2026-10-13 12:25

来源机构：Hugging Face