Training foundation models, such as GPT-3 and PaLM, can be extremely expensive, often involving tens of thousands of GPUs running continuously for months. These models are typically trained in specialized clusters featuring fast, homogeneous interconnects and using carefully designed software systems that support both data parallelism and model/pipeline parallelism. Such dedicated clusters can be costly and difficult to obtain. Can we instead leverage the much greater amount of decentralized, heterogeneous, and lower-bandwidth interconnected compute? Previous works examining the heterogeneous, decentralized setting focus on relatively small models that can be trained in a purely data parallel manner. State-of-the-art schemes for model parallel foundation model training, such as Megatron, only consider the homogeneous data center setting. In this paper, we present the first study of training large foundation models with model parallelism in a decentralized regime over a heterogeneous network. Our key technical contribution is a scheduling algorithm that allocates different computational "tasklets" in the training of foundation models to a group of decentralized GPU devices connected by a slow heterogeneous network. We provide a formal cost model and further propose an efficient evolutionary algorithm to find the optimal allocation strategy. We conduct extensive experiments that represent different scenarios for learning over geo-distributed devices simulated using real-world network measurements. In the most extreme case, across 8 different cities spanning 3 continents, our approach is 4.8X faster than prior state-of-the-art training systems (Megatron).

翻译：训练基础模型（如GPT-3和PaLM）成本极高，通常需要数万块GPU连续运行数月。这些模型通常在专用集群中训练，配备高速同构互连，并使用精心设计的支持数据并行和模型/流水线并行的软件系统。此类专用集群成本高昂且难以获取。我们能否利用数量更多、去中心化、异构且带宽较低的互联计算资源？以往针对异构去中心化场景的研究主要关注可通过纯数据并行方式训练的较小模型。而最先进的模型并行基础模型训练方案（如Megatron）仅考虑同构数据中心环境。本文首次研究了在异构网络去中心化环境下使用模型并行训练大规模基础模型的问题。我们的关键技术贡献是一种调度算法，该算法将基础模型训练中的不同计算"任务块"分配给通过低速异构网络连接的分布式GPU设备组。我们提出了形式化成本模型，并进一步设计了一种高效的进化算法来寻找最优分配策略。我们使用真实网络测量数据模拟了跨地理分布式设备学习的多种场景，并开展了大量实验。在最极端情况下（跨越三大洲8个不同城市），我们的方法比现有最先进的训练系统（Megatron）快4.8倍。