Large language models (LLMs) have demonstrated remarkable success as foundational models, benefiting various downstream applications through fine-tuning. Recent studies on loss scaling have demonstrated the superior performance of larger LLMs compared to their smaller counterparts. Nevertheless, training LLMs with billions of parameters poses significant challenges and requires considerable computational resources. For example, training a one trillion parameter GPT-style model on 20 trillion tokens requires a staggering 120 million exaflops of computation. This research explores efficient distributed training strategies to extract this computation from Frontier, the world's first exascale supercomputer dedicated to open science. We enable and investigate various model and data parallel training techniques, such as tensor parallelism, pipeline parallelism, and sharded data parallelism, to facilitate training a trillion-parameter model on Frontier. We empirically assess these techniques and their associated parameters to determine their impact on memory footprint, communication latency, and GPU's computational efficiency. We analyze the complex interplay among these techniques and find a strategy to combine them to achieve high throughput through hyperparameter tuning. We have identified efficient strategies for training large LLMs of varying sizes through empirical analysis and hyperparameter tuning. For 22 Billion, 175 Billion, and 1 Trillion parameters, we achieved GPU throughputs of $38.38\%$, $36.14\%$, and $31.96\%$, respectively. For the training of the 175 Billion parameter model and the 1 Trillion parameter model, we achieved $100\%$ weak scaling efficiency on 1024 and 3072 MI250X GPUs, respectively. We also achieved strong scaling efficiencies of $89\%$ and $87\%$ for these two models.

翻译：大型语言模型（LLM）作为基础模型已展现出卓越的成功，通过微调使各类下游应用受益。近期关于损失缩放的研究表明，更大规模的LLM相比其较小版本具有更优性能。然而，训练拥有数十亿参数的LLM面临重大挑战，需要大量计算资源。例如，在20万亿个token上训练一个万亿参数的GPT风格模型需要惊人的1.2亿百亿亿次浮点运算。本研究探索高效分布式训练策略，以在专为开放科学打造的世界首台百亿亿级超级计算机Frontier上提取这些计算能力。我们启用并研究了多种模型并行与数据并行训练技术，如张量并行、流水线并行和分片数据并行，以支持在Frontier上训练万亿参数模型。我们通过实验评估这些技术及其关联参数，确定其对内存占用、通信延迟和GPU计算效率的影响。我们分析了这些技术间的复杂相互作用，并通过超参数调优找到组合策略以实现高吞吐量。通过实证分析与超参数调优，我们确定了训练不同规模大型LLM的高效策略。对于220亿、1750亿及1万亿参数模型，我们分别实现了38.38%、36.14%和31.96%的GPU吞吐量。在1750亿参数模型和1万亿参数模型的训练中，我们分别在1024块和3072块MI250X GPU上实现了100%的弱扩展效率。此外，这两个模型的强扩展效率分别达到89%和87%。