混合专家语言模型可成为强大高效的检索器
原标题：Mixture-of-Experts Language Models Can Be Strong and Efficient Retrievers
AI 摘要
该研究系统比较了混合专家（MoE）与稠密LLM作为一阶段检索器的效果与效率。在相同训练流程下，MoE检索器在BEIR上比激活参数量相当的稠密模型高1.4–3.0 nDCG@10，其中一个MoE检索器以少59%的激活参数和低18%的查询编码时间匹配8B稠密检索器。研究还表明，无需重训练或重建索引即可减少查询编码使用的专家数，保留99%以上效果并将查询编码时间最多降低26%。
正文节选
Mixture-of-Experts Language Models Can Be Strong and Efficient Retrievers Abstract Recent work has shown that fine-tuning decoder-only large language models (LLMs) for retrieval yields strong first-stage retrievers, with effectiveness improving as backbones grow in size. However, every query and document must pass through the full model, so encoding cost increases with model size. Mixture-of-Experts (MoE) LLMs activate only a subset of parameters per token and are widely used to scale generative