Pretraining Large Language Models (LLMs) from scratch requires massive amount of compute. Aurora super computer is an ExaScale machine with 127,488 Intel PVC (Ponte Vechio) GPU tiles. In this work, we showcase LLM pretraining on Aurora at the scale of 1000s of GPU tiles. Towards this effort, we developed Optimus, an inhouse training library with support for standard large model training techniques. Using Optimus, we first pretrained Mula-1B, a 1 Billion dense model and Mula-7B-A1B, a 7 Billion Mixture of Experts (MoE) model from scratch on 3072 GPU tiles for the full 4 trillion tokens of the OLMoE-mix-0924 dataset. We then demonstrated model scaling by pretraining three large MoE models Mula-20B-A2B, Mula-100B-A7B, and Mula-220B-A10B till 100 Billion tokens on the same dataset. On our largest model Mula-220B-A10B, we pushed the compute scaling from 384 to 12288 GPU tiles and observed scaling efficiency of around 90% at 12288 GPU tiles. We significantly improved the runtime performance of MoE models using custom GPU kernels for expert computation, and a novel EP-Aware sharded optimizer resulting in training speedups up to 1.71x. As part of the Optimus library, we also developed a robust set of reliability and fault tolerant features to improve training stability and continuity at scale.

翻译：从头开始预训练大语言模型需要海量计算资源。Aurora超级计算机是一台百亿亿次级机器，配备127,488块Intel PVC（Ponte Vechio）GPU计算单元。本研究展示了在数千块GPU计算单元规模的Aurora平台上进行大语言模型预训练。为此，我们开发了Optimus——一个支持标准大型模型训练技术的内部训练库。利用Optimus，我们首先在3072块GPU计算单元上，基于OLMoE-mix-0924数据集的完整4万亿词元，从头预训练了Mula-1B（10亿参数稠密模型）和Mula-7B-A1B（70亿参数混合专家模型）。随后，我们通过在同一数据集上预训练三个大型MoE模型（Mula-20B-A2B、Mula-100B-A7B和Mula-220B-A10B）直至1000亿词元，展示了模型扩展性。在最大模型Mula-220B-A10B上，我们将计算规模从384块GPU计算单元扩展至12288块，并在12288块GPU计算单元上观察到约90%的扩展效率。通过使用针对专家计算定制的GPU内核以及新型EP感知分片优化器，我们显著提升了MoE模型的运行时性能，训练加速比最高达1.71倍。作为Optimus库的组成部分，我们还开发了一套稳健的可靠性与容错特性，以提升大规模训练过程中的稳定性和连续性。