流形投影与迭代自编码器精炼用于掩码语言建模
原标题：Manifold Projection and Iterative Autoencoder Refinement for Masked Language Modeling
AI 摘要
该论文提出用低秩瓶颈自编码器替代Transformer中的自注意力机制，构建了WindowMixAE、GlobalMixAE和ChannelMixAE三个混合模块，分别处理局部邻域、全序列和注意力头之间的信息混合。在掩码位置预测上，作者引入迭代精炼过程，包含向邻居加权平均拉近的pulling步骤和通过自编码器投影回学习流形的correcting步骤。在C4上预训练并与参数匹配的BERT基线对比，该架构以约更少的FLOPs达到注意力机制相当部分性能，并在稀有词频段上通过频率感知训练调度追平BERT和TinyBERT。
正文节选
Manifold Projection and Iterative Autoencoder Refinement for Masked Language Modeling Abstract In Transformer-based masked language models, attention is the primary mechanism for context mixing, but there are other ways to mix data across tokens. Recent attention-free mixers replace attention with fixed or hypernetwork-generated MLPs, alternating their dynamic, content-dependent weighting for computational simplicity. We build an alternative that gets the same property from a low-rank bottleneck