稠密模型与 MoE 模型对比：激活参数、吞吐量与选择时机
原标题：Dense vs. MoE Models: Active Parameters, Throughput, and When to Choose Each
AI 摘要
NVIDIA 技术博客对比了稠密模型与 MoE（混合专家）架构，解释 MoE 如何让 30B 参数模型每 token 仅激活约 3B 参数，并仍利用大模型容量。文章以 Nemotron 3.5 Lightning 为例，说明 MoE 通过路由网络为每个 token 选择部分专家 FFN，注意力与嵌入权重仍全量参与。核心结论是 MoE 将显存成本（总参数）与计算成本（激活参数）解耦，在低并发下吞吐优势明显，但高并发时路由与显存搬运会缩小其延迟优势，且所有专家需常驻 GPU 显存，挤占 KV cache 空间。
正文节选
How can a 30B-parameter model activate only 3B parameters per token, and still use the capacity of the larger model? Nemotron 3.5 Lightning illustrates the answer: It uses a Mixture-of-Experts (MoE) architecture that selects only a subset of its parameters for each token. There are two dominant model architectures: Dense model and MoE. How a model organizes its parameters matters as much as how many it has. The right choice affects throughput, memory cost, and serving complexity more than raw pa