Training foundation models, such as GPT-3 and PaLM, can be extremely expensive, often involving tens of thousands of GPUs running continuously for months. These models are typically trained in specialized clusters featuring fast, homogeneous interconnects and using carefully designed software systems that support both data parallelism and model/pipeline parallelism. Such dedicated clusters can be costly and difficult to obtain. Can we instead leverage the much greater amount of decentralized, heterogeneous, and lower-bandwidth interconnected compute? Previous works examining the heterogeneous, decentralized setting focus on relatively small models that can be trained in a purely data parallel manner. State-of-the-art schemes for model parallel foundation model training, such as Megatron, only consider the homogeneous data center setting. In this paper, we present the first study of training large foundation models with model parallelism in a decentralized regime over a heterogeneous network. Our key technical contribution is a scheduling algorithm that allocates different computational "tasklets" in the training of foundation models to a group of decentralized GPU devices connected by a slow heterogeneous network. We provide a formal cost model and further propose an efficient evolutionary algorithm to find the optimal allocation strategy. We conduct extensive experiments that represent different scenarios for learning over geo-distributed devices simulated using real-world network measurements. In the most extreme case, across 8 different cities spanning 3 continents, our approach is 4.8X faster than prior state-of-the-art training systems (Megatron).

翻译：训练基础模型（如GPT-3和PaLM）成本极其高昂，通常需要数万个GPU连续运行数月。这类模型通常在专用集群中训练，配备快速同构互连网络，并使用精心设计的软件系统来支持数据并行与模型/流水线并行。此类专用集群成本高昂且难以获取。我们能否利用更广泛的去中心化、异构、低带宽互联计算资源？先前针对异构去中心化环境的研究聚焦于可通过纯数据并行方式训练的较小模型。而当前最先进的模型并行基础模型训练方案（如Megatron）仅考虑同构数据中心场景。本文首次研究了在异构网络去中心化环境下使用模型并行训练大型基础模型的问题。我们的核心技术贡献是一种调度算法，可将基础模型训练中的不同计算"任务块"分配给由慢速异构网络连接的分散式GPU设备群。我们建立了形式化的成本模型，并进一步提出了一种高效演化算法来寻找最优分配策略。基于真实网络测量数据模拟的地理分布式设备学习场景中，我们开展了广泛实验。在最极端情况下（横跨三大洲8个城市），我们的方法比现有最先进的训练系统（Megatron）快4.8倍。