NVIDIA 开源集群级训练自动恢复服务 NVIDIA Recovery
原标题：Keep Distributed Training Running with Automated Workload Recovery
AI 摘要
NVIDIA 发布开源的工作负载级恢复服务 NVIDIA Recovery，属于 NVIDIA DSX OS 软件栈，可自动检测分布式训练中断、归因故障资源，并协调隔离与从检查点重启。该服务与 Slurm、NVSentinel、NVRx 等组件集成，在集群层面处理并发故障，减少人工干预并维持训练 goodput。NVIDIA 披露在一个训练集群上日均覆盖近 200 个作业，检测与恢复动作平均延迟分别为 69 秒和 94 秒。
正文节选
As distributed AI training scales, failures across hardware and software dependencies increasingly disrupt training progress, making fast, reliable recovery essential. A stalled training rank can trigger an NVIDIA Collective Communications Library (NCCL) timeout, a GPU node can stop responding to the scheduler, or a straggler can slow the entire job. Each disruption is manageable on its own, but the cumulative impact and recovery overhead determine how efficiently the system converts wall-clock