Maglev：滑动循环记忆提升长上下文建模

AI 摘要

Maglev 是一种新型循环 Transformer 架构，通过固定大小记忆推广滑动窗口注意力，同时保持训练可并行化。它由预填充器 Q 和解码器 P 两个耦合模型组成，使用记忆一致性损失对齐两者，使推理时仅用 P 即可。实验表明，Maglev 在验证损失和下游预训练基准上优于滑动窗口和潜在循环 Transformer 基线，且参数共享可减少内存占用。

正文节选

Maglev: Sliding Recurrent Memory Abstract A recurrent Transformer with fixed-size memory and coupled prefiller-decoder training improves long-context modeling while enabling efficient parallel training and reduced inference cost. We introduce , a recurrent Transformer architecture with fixed-size memory that generalizes sliding-window attention while remaining parallelizable during training. consists of two coupled models: a prefiller Q, which leverages full attentionIn practice, we use interlea

发布时间：—

抓取时间：2026-08-15 07:37

来源机构：Hugging Face