Training and deploying large machine learning (ML) models is time-consuming and requires significant distributed computing infrastructures. Based on real-world large model training on datacenter-scale infrastructures, we show 14~32% of all GPU hours are spent on communication with no overlapping computation. To minimize the outstanding communication latency, in this work, we develop an agile performance modeling framework to guide parallelization and hardware-software co-design strategies. Using the suite of real-world large ML models on state-of-the-art GPU training hardware, we demonstrate 2.24x and 5.27x throughput improvement potential for pre-training and inference scenarios, respectively.

翻译：训练和部署大规模机器学习（ML）模型耗时巨大，需要大规模分布式计算基础设施。基于数据中心级基础设施上的真实大规模模型训练，我们发现14%~32%的GPU计算时间消耗在无重叠计算的通信上。为最小化显著的通信延迟，本研究开发了一种敏捷性能建模框架，用于指导并行化策略与软硬件协同设计。通过使用最先进的GPU训练硬件对真实大规模ML模型进行评估，我们证明了预训练和推理场景分别具有2.24倍和5.27倍的吞吐量提升潜力。