NVIDIA Attribution：诊断分布式 AI 工作负载故障
原标题：Diagnose Distributed AI Workload Failures with NVIDIA Attribution
AI 摘要
NVIDIA 发布技术博客，介绍其用于诊断分布式 AI 工作负载故障的 NVIDIA Attribution 工具。该工具将工作负载范围内的日志、指标、遥测和调度器状态关联起来，输出包含结果、根因、受影响节点、症状和证据链的结构化诊断。文章通过一个多卡训练任务中 XID 149 硬件故障引发 NCCL 错误的示例，演示了从证据提取到根因判定的完整流程。该工具已在 NVIDIA 生产环境中使用多年，每周分析数十万个 GPU 上的数百万个工作负载。
正文节选
In distributed AI workloads, a single error can produce a cascade of symptoms. A GPU or fabric event may surface as NCCL errors on other ranks, scheduler state changes, and thousands of downstream messages. The reported exception shows where the workload stopped, not where the failure began. In this post, you’ll learn how to run NVIDIA Attribution on sample workload evidence, inspect the diagnosis, and explore how attribution rules identify failure patterns. NVIDIA Attribution correlates workloa