LoKiFormer：局部感知注意力与解耦知识记忆提升大模型预训练效率
原标题：LoKiFormer: Locality-aware Attention with Decoupled Knowledge Memory for Efficient Large Language Model Pretraining
AI 摘要
LoKiFormer 是一种新的大语言模型架构，通过引入局部融合注意力（LFA）和解耦知识记忆模块（KMM）来提升预训练效率。LFA 利用卷积融合显式捕捉局部模式，KMM 通过参数化键值记忆将知识存储与计算解耦，实现直接知识检索。实验表明，LoKiFormer 的预训练收敛速度比基线模型快 1.33 倍。
正文节选
LoKiFormer: Locality-aware Attention with Decoupled Knowledge Memory for Efficient Large Language Model Pretraining Abstract Large language models (LLMs) have achieved remarkable breakthroughs across various applications. However, their architectures remain inefficient in pretraining due to two main limitations: (i) self-attention lacks an explicit inductive bias for locality, leading to redundant modeling of sequence-internal local information; (ii) mixture-of-experts (MoE) implicitly couples k