The rapid growth of large-scale machine learning (ML) has made distributed training across multiple GPUs a fundamental component of modern ML systems. As model sizes and computational throughput continue to increase, communication overhead has become a dominant bottleneck in multi-GPU training, particularly when computation and communication are executed sequentially. This work explores concurrent execution of computation and collective communication using two portable runtime controls: shared-memory-driven occupancy shaping for computation kernels and elevated scheduling priority for communication kernels. Our approach regulates computation-kernel residency through per-block shared-memory allocation, leaving sufficient on-chip resources for communication kernels to make progress. In addition, assigning higher priority to communication streams ensures steady communication progress once resources become available. Experiments on NVIDIA A40, A100, H100, and AMD MI250X GPUs demonstrate that the proposed method enables effective computation-communication overlap and reduces total execution time by up to 25.5 percent, without modifying vendor libraries or kernel implementations.

翻译：大规模机器学习的迅速发展使得跨多GPU分布式训练成为现代机器学习系统的核心组成部分。随着模型规模和计算吞吐量的持续增长，通信开销已成为多GPU训练的主要瓶颈，特别是在计算与通信顺序执行的情况下。本文利用两种可移植运行时控制机制实现了计算与集合通信的并发执行：面向计算内核的共享内存驱动驻留数调控技术，以及面向通信内核的高调度优先级策略。该方法通过按块分配共享内存来调控计算内核的驻留空间，为通信内核保留足够的片上资源以持续推进。此外，为通信流分配更高优先级可确保资源可用时通信稳定推进。在NVIDIA A40、A100、H100及AMD MI250X GPU上的实验表明，所提方法能够实现高效的计算-通信重叠，在无需修改厂商库或内核实现的前提下，将总执行时间最多降低25.5%。