在 Amazon EKS 上使用 NVRx 实现容错分布式训练
原标题：Fault tolerant distributed training on Amazon EKS using NVRx
AI 摘要
AWS Machine Learning Blog 发布技术文章，介绍如何在 Amazon EKS 上结合 NVIDIA Resiliency Extension (NVRx) 实现容错的 PyTorch FSDP 分布式训练。方案通过异步检查点、进程内重启和 ft_launcher 作业内重启三层机制，分别应对软故障、硬故障和节点丢失，并在 H100 GPU 上从 2 节点到 8 节点规模进行了基准测试。文章指出同步检查点最多占用 40% 总墙钟时间，而该方案可减少 GPU 空闲并加速故障恢复。
正文节选
Fault tolerant distributed training on Amazon EKS using NVRx Large-scale distributed training jobs run for hours or days across dozens of nodes. At that scale and duration, interruptions are statistically inevitable: network partitions, memory errors, software exceptions, or infrastructure events will eventually disrupt at least one worker. A single GPU fault triggers a cascade: NVIDIA Collective Communication Library (NCCL) timeouts propagate to healthy workers, pods crash and restart out of sy