BCMT：块级因果记忆Transformer实现高效长上下文建模
原标题：BCMT: Blockwise Causal Memory Transformer
AI 摘要
BCMT（Blockwise Causal Memory Transformer）是一种新型Transformer架构，通过将局部token交互与全局上下文传播解耦，利用指数因果记忆机制高效建模长距离依赖。实验表明，在长达1024 token的语言建模任务中，BCMT在验证性能上与密集Transformer相当，同时显著提升训练吞吐量并降低内存消耗。该架构无需全局注意力或学习记忆状态，且完全可并行化，为长上下文语言建模提供了有效替代方案。
正文节选
BCMT: Blockwise Causal Memory Transformer Abstract Transformer architectures rely on dense self-attention to model long-range dependencies, but this mechanism exhibits quadratic complexity with respect to sequence length. We introduce BCMT (Blockwise Causal Memory Transformer), an architecture for long-context language modeling that decouples local token interactions from global context propagation. Dense causal self-attention is applied independently within local blocks, while each block produc