The rapid growth in the size of large language models has necessitated the partitioning of computational workloads across accelerators such as GPUs, TPUs, and NPUs. However, these parallelization strategies incur substantial data communication overhead significantly hindering computational efficiency. While communication-computation overlap presents a promising direction, existing data slicing based solutions suffer from tail latency. To overcome this limitation, this research introduces a novel communication-computation overlap technique to eliminate this tail latency in state of the art overlap methods for distributed LLM training. The aim of this technique is to effectively mitigate communication bottleneck of tensor parallelism and data parallelism for distributed training and inference. In particular, we propose a novel method termed Flash-Overlap that replaces conventional collective operations of reduce-scatter and all-gather with decomposed peer-to-peer (P2P) communication and schedules partitioned computations to enable fine-grained overlap. Our method provides an exact algorithm for reducing communication overhead that eliminates tail latency. Moreover, it presents a versatile solution compatible with data-parallel training and various tensor-level parallelism strategies, including TPSP and UP. Experimental evaluations demonstrate that our technique consistently achieves lower latency, superior Model FLOPS Utilization (MFU), and high throughput.

翻译：大规模语言模型规模的快速增长要求将计算工作负载划分至GPU、TPU和NPU等加速器上。然而，这些并行化策略会引入大量数据通信开销，显著降低计算效率。尽管通信-计算重叠技术展现出良好前景，但现有基于数据切分的解决方案存在尾延迟问题。为突破这一局限，本研究提出一种新型通信-计算重叠技术，可消除当前最先进的分布式大语言模型训练重叠方法中的尾延迟。该技术旨在有效缓解张量并行与数据并行在分布式训练与推理中的通信瓶颈。具体而言，我们提出名为Flash-Overlap的创新方法，通过将传统的reduce-scatter与all-gather集体操作分解为点对点（P2P）通信，并调度分区计算实现细粒度重叠。该方法提供精确算法降低通信开销，完全消除尾延迟。此外，其通用性解决方案兼容数据并行训练及包括TPSP和UP在内的多种张量级并行策略。实验评估表明，本技术持续实现更低延迟、更优模型算力利用率与高吞吐量。