泰勒级数展开实现固定循环状态的线性时间Transformer
原标题：Expanding Attention: A Taylor-Series Route to Linear-Time Transformers with a Fixed Number of Recurrent States
AI 摘要
该论文提出将注意力中的指数核展开为截断泰勒级数，并将每一项分解为张量幂内积，从而把因果历史压缩为固定数量的循环状态，每个状态每 token 仅需一次秩一外积更新，与上下文长度无关。该方法在截断核意义下无损，状态可跨块相加，因此预填充支持并行扫描。实测显示每 token 解码延迟恒定，精度随阶数指数提升但成本组合增长，最优为二阶，且查询和键归一化是必要前提。
正文节选
Expanding Attention: A Taylor-Series Route to Linear-Time Transformers with a Fixed Number of Recurrent States Abstract We expand the exponential kernel of attention into a truncated Taylor series of terms and factorize each term into a tensor-power inner product, which collapses the causal history into exactly recurrent states, each updated by one rank-one outer product per token and independent of context length . The recursion is lossless with respect to the truncated kernel ( in float64), a