Large language models (LLMs) such as GPT-3, OPT, and LLaMA have demonstrated remarkable accuracy in a wide range of tasks. However, training these models can incur significant expenses, often requiring tens of thousands of GPUs for months of continuous operation. Typically, this training is carried out in specialized GPU clusters equipped with homogeneous high-speed Remote Direct Memory Access (RDMA) network interface cards (NICs). The acquisition and maintenance of such dedicated clusters is challenging. Current LLM training frameworks, like Megatron-LM and Megatron-DeepSpeed, focus primarily on optimizing training within homogeneous cluster settings. In this paper, we introduce Holmes, a training framework for LLMs that employs thoughtfully crafted data and model parallelism strategies over the heterogeneous NIC environment. Our primary technical contribution lies in a novel scheduling method that intelligently allocates distinct computational tasklets in LLM training to specific groups of GPU devices based on the characteristics of their connected NICs. Furthermore, our proposed framework, utilizing pipeline parallel techniques, demonstrates scalability to multiple GPU clusters, even in scenarios without high-speed interconnects between nodes in distinct clusters. We conducted comprehensive experiments that involved various scenarios in the heterogeneous NIC environment. In most cases, our framework achieves performance levels close to those achievable with homogeneous RDMA-capable networks (InfiniBand or RoCE), significantly exceeding training efficiency within the pure Ethernet environment. Additionally, we verified that our framework outperforms other mainstream LLM frameworks under heterogeneous NIC environment in terms of training efficiency and can be seamlessly integrated with them.

翻译：诸如GPT-3、OPT和LLaMA等大型语言模型（LLMs）在各类任务中展现出卓越的准确性。然而，训练这些模型需要投入巨额成本，往往需要数万块GPU连续运行数月。通常，这种训练在配备同构高速远程直接内存访问（RDMA）网卡（NICs）的专用GPU集群中进行。获取和维护此类专属集群颇具挑战性。当前的LLM训练框架（如Megatron-LM和Megatron-DeepSpeed）主要专注于优化同构集群环境下的训练。本文提出了Holmes，一个针对异构NIC环境的LLM训练框架，该框架采用精心设计的数据和模型并行策略。我们的核心技术贡献在于一种新颖的调度方法，该方法根据LLM训练中不同计算任务子集的GPU设备所连接NIC的特性，智能地为其分配任务。此外，我们提出的框架利用流水线并行技术，能够扩展到多个GPU集群，即便在不同集群节点之间缺乏高速互连的场景下也能有效运行。我们进行了涵盖异构NIC环境中多种场景的全面实验。在大多数情况下，我们的框架实现了接近同构RDMA网络（InfiniBand或RoCE）性能的水平，显著优于纯以太网环境下的训练效率。此外，我们验证了在异构NIC环境下，该框架在训练效率上优于其他主流LLM框架，并且能够与其无缝集成。