Google DeepMind 推出 Decoupled DiLoCo，实现跨区域高效分布式训练
原标题：Decoupled DiLoCo: A new frontier for resilient, distributed AI training
AI 摘要
Google DeepMind 推出了 Decoupled DiLoCo，一种新型分布式 AI 训练方法，通过将通信融入计算周期，避免同步阻塞，从而在跨区域网络上实现高效训练。他们成功使用 2-5 Gbps 的广域网在四个美国区域训练了 120 亿参数模型，速度比传统同步方法快 20 倍以上。该方法还支持混合不同代际的硬件（如 TPU v6e 和 TPU v5p），在保持性能的同时延长硬件寿命并增加可用算力。
正文节选
Decoupled DiLoCo is not only more resilient to failures, but is also practical for executing production-level, fully distributed pre-training. We successfully trained a 12 billion parameter model across four separate U.S. regions using 2-5 Gbps of wide-area networking (a level relatively achievable using existing internet connectivity between datacenter facilities, rather than requiring new custom network infrastructure between facilities). Notably, the system achieved this training result more