Hugging Face 发布 Olmo-core 3：面向大型 MoE 的开放可扩展训练基础设施
原标题：Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEs
AI 摘要
Hugging Face 发布 Olmo-core 3，这是其大模型训练框架的重大升级，核心是重新设计的开放 MoE 训练系统，可扩展到万亿参数规模。它从 FSDP 切换到基于 DDP 的系统，结合专家并行、流水线并行、分布式优化器以及 MXFP8 低精度格式等优化。在 8 张 NVIDIA B300 GPU 上，470 亿参数 MoE 吞吐量达每 GPU 每秒 5.2 万 token，约为旧实现的 2.7 倍；并在 512 张 GPU 上基准测试了 1.2 万亿参数模型。
正文节选
Today we’re releasing Olmo-core 3, a significant upgrade to our framework for developing large language models featuring a redesigned open mixture-of-experts (MoE) training system. Olmo-core 3 is designed to scale MoE training into the trillion-parameter range while preserving computational efficiency. It’s one of the core systems behind the next generation of Olmo, and part of our ongoing commitment to open up the tools and training infrastructure behind each new model. Training large AI models