Various distributed deep neural network (DNN) training technologies lead to increasingly complicated use of collective communications on GPU. The deadlock-prone collectives on GPU force researchers to guarantee that collectives are enqueued in a consistent order on each GPU to prevent deadlocks. In complex distributed DNN training scenarios, manual hardcoding is the only practical way for deadlock prevention, which poses significant challenges to the development of artificial intelligence. This paper presents OCCL, which is, to the best of our knowledge, the first deadlock-free collective communication library for GPU supporting dynamic decentralized preemption and gang-scheduling for collectives. Leveraging the preemption opportunity of collectives on GPU, OCCL dynamically preempts collectives in a decentralized way via the deadlock-free collective execution framework and allows dynamic decentralized gang-scheduling via the stickiness adjustment scheme. With the help of OCCL, researchers no longer have to struggle to get all GPUs to launch collectives in a consistent order to prevent deadlocks. We implement OCCL with several optimizations and integrate OCCL with a distributed deep learning framework OneFlow. Experimental results demonstrate that OCCL achieves comparable or better latency and bandwidth for collectives compared to NCCL, the state-of-the-art. When used in distributed DNN training, OCCL can improve the peak training throughput by up to 78% compared to statically sequenced NCCL, while introducing overheads of less than 6.5% across various distributed DNN training approaches.

翻译：各种分布式深度神经网络训练技术导致GPU上集合通信的使用日益复杂。GPU上容易发生死锁的集合操作迫使研究者必须确保每个GPU以一致顺序入队集合操作，从而防止死锁。在复杂的分布式DNN训练场景中，手动硬编码是唯一实用的死锁预防方式，这对人工智能的发展构成了重大挑战。本文提出OCCL——据我们所知，这是首个支持动态分散抢占和集合分组调度的无死锁GPU集合通信库。通过利用GPU上集合操作的抢占机会，OCCL借助无死锁集合执行框架以分散方式动态抢占集合操作，并利用粘性调整方案实现动态分散分组调度。借助OCCL，研究者不再需要费心让所有GPU以一致顺序启动集合操作来防止死锁。我们通过多项优化实现了OCCL，并将其与分布式深度学习框架OneFlow集成。实验结果表明，与当前最先进的NCCL相比，OCCL在集合操作延迟和带宽方面表现相当或更优。在分布式DNN训练中，与静态排序的NCCL相比，OCCL可将峰值训练吞吐量提升高达78%，同时在各种分布式DNN训练方法中引入的开销低于6.5%。