Netflix 将 GPU 可靠性视为分布式系统问题
原标题：Netflix Treats GPU Reliability as a Distributed Systems Problem
AI 摘要
Netflix 详细介绍了其构建的大规模 GPU 训练与推理可靠性平台，该平台基于 Kubernetes，采用分层健康检查、自动修复和容错训练机制，使长周期训练任务在硬件故障下仍能持续运行。Netflix 报告 GPU 节点可用性超过 99.95%，任务成功率从约 98% 提升至 99.5% 以上。文章还对比了 Meta 在 Llama 3 训练中硬件故障占训练中断超 66% 的经验，指出在数千 GPU 规模下可靠性需按分布式系统问题处理。
正文节选
Netflix has detailed how it is building a reliability platform for large-scale GPU training and inference, using layered health checks, automated remediation, and fault-tolerant training to keep jobs running despite hardware failures. The problem is particularly acute for large AI training jobs. A model may run for weeks across hundreds of GPUs, with all workers operating in lockstep. A single failed GPU, degraded network adapter, or slow machine can therefore stop or significantly slow the enti