动态稀疏MoE的分布一致性推理
原标题：Distribution-Consistent Inference for Dynamic Sparse Mixture-of-Experts
AI 摘要
该论文研究动态稀疏专家混合（SMoE）推理中减少激活专家数量带来的分布偏移问题。作者发现减少激活专家会持续增大SMoE输出的RMS尺度和方差，导致表示不匹配，进而造成下游性能下降，而这一损失不仅源于专家容量减少。为此提出层间分布对齐（LDA），一种无需重训练的轻量级推理时校正方法，利用逐层校准统计量对齐降路由表示与默认配置，在多个SMoE大模型和基准上恢复了大部分性能损失，且开销可忽略。
正文节选
Distribution-Consistent Inference for Dynamic Sparse Mixture-of-Experts Abstract Mixture-of-Experts (MoE) architectures have emerged as a powerful paradigm for scaling model capacity while preserving efficient inference in large foundation models. However, most MoE models use a fixed top- expert selection policy, assigning the same expert budget to every token even when fewer experts may be sufficient. Inference-time dynamic top- routing can reduce computation without retraining, but existing me