非对称注意力头：Transformer注意力的结构化头级上下文分配
原标题：Asymmetric Attention Heads: Structured Head-Wise Context Allocation for Transformer Attention
AI 摘要
本文提出非对称注意力头（AAH）框架，允许Transformer各头或头组使用不同长度的因果上下文窗口，同时保持标准多头注意力输出接口。在4096 token的实验中，多种AAH局部分配变体取得了比纯全注意力更低的验证损失。作者强调AAH主要作为质量与可解释性的结构化上下文分配机制，当前实现未证明GPU FLOPs减少。
正文节选
Asymmetric Attention Heads: Structured Head-Wise Context Allocation for Transformer Attention Abstract Standard multi-head attention (MHA) gives every head the same full causal context span, although heads can serve different contextual roles. Some heads may rely mainly on nearby lexical or syntactic context, while others may depend on longer-range relations such as entity interactions, discourse links, or state changes. We present Asymmetric Attention Heads (AAH), a head-wise context-allocation