注意力感知路由：在 MoE 中耦合路由与注意力

AI 摘要

该论文提出 Attention-Aware Routing (AAR)，在 MoE 语言模型中用滑动窗口的注意力权重为路由器补充上下文信息，同时冻结基础 Transformer 只训练路由参数。在 OLMoE 上，AAR 相比仅路由 SFT 基线提升了 GSM8K 数学推理表现，并揭示路由与注意力构成耦合回路：路由变化经残差流放大注意力汇。研究还发现 AAR 具有深度敏感性，浅层应用可能损害事实检索，而深层应用可保持数学推理增益，体现检索与推理的深度权衡。

正文节选

Attention-Aware Routing: Coupling Routing and Attention in MoEs Abstract In Mixture-of-Experts language models, the router typically selects and weights experts based on the token’s hidden state, utilizing limited contextual information. We propose Attention-Aware Routing (AAR), which augments the router with temporal and spectral features extracted from a sliding window of attention weights that represent a summary of the model’s contextual state, disentangled from the hidden state. Keeping the

发布时间：2026-09-22 12:00

抓取时间：2026-09-21 12:12

来源机构：arXiv