Many deep learning applications benefit from using large models with billions of parameters. Training these models is notoriously expensive due to the need for specialized HPC clusters. In this work, we consider alternative setups for training large models: using cheap "preemptible" instances or pooling existing resources from multiple regions. We analyze the performance of existing model-parallel algorithms in these conditions and find configurations where training larger models becomes less communication-intensive. Based on these findings, we propose SWARM parallelism, a model-parallel training algorithm designed for poorly connected, heterogeneous and unreliable devices. SWARM creates temporary randomized pipelines between nodes that are rebalanced in case of failure. We empirically validate our findings and compare SWARM parallelism with existing large-scale training approaches. Finally, we combine our insights with compression strategies to train a large Transformer language model with 1B shared parameters (approximately 13B before sharing) on preemptible T4 GPUs with less than 200Mb/s network.

翻译：许多深度学习应用受益于使用具有数十亿参数的大模型。由于对专用高性能计算集群的需求，训练这些模型成本高昂。在本工作中，我们考虑了大模型训练的可替代方案：使用廉价的“可抢占”实例或整合来自多个区域的现有资源。我们分析了现有模型并行算法在这些条件下的性能，并发现了训练更大模型时通信开销降低的配置。基于这些发现，我们提出了SWARM并行——一种针对弱连接、异构且不可靠设备设计的模型并行训练算法。SWARM算法在节点间创建临时随机化流水线，可在发生故障时重新平衡。我们通过实验验证了我们的发现，并将SWARM并行与现有大规模训练方法进行了比较。最后，我们将研究成果与压缩策略相结合，在可抢占的T4 GPU（网络带宽低于200Mb/s）上成功训练了一个具有10亿共享参数（共享前约130亿参数）的大型Transformer语言模型。