数据驱动并行：高效训练可变长序列的新方法
原标题：Training Variable Long Sequences with Data-Centric Parallel
AI 摘要
arXiv 论文提出 Data-Centric Parallel (DCP)，一种用于训练可变长序列的分布式并行方法。DCP 通过根据每个 batch 的序列长度动态调整并行大小、梯度累积和重计算等运行时设置，在 32 块 H200 GPU 上实现了最高 2.88 倍的加速，且仅需 10 行代码即可集成到任意模型中。该方法旨在打破现有方法在效率和易用性之间的权衡，为可变长序列分布式训练提供简单有效的基线。
正文节选
Training Variable Long Sequences with Data-Centric Parallel Abstract Training deep learning models on variable long sequences poses significant computational challenges. Existing methods force a difficult trade-off between efficiency and ease-of-use. Simple approaches use static configurations that cause workload imbalance low efficiency, while complex methods introduces significant complexity and code change for new models. To break this trade-off, we introduce Data-Centric Parallel (DCP). Its