Distributed deep learning (DL) has become prevalent in recent years to reduce training time by leveraging multiple computing devices (e.g., GPUs/TPUs) due to larger models and datasets. However, system scalability is limited by communication becoming the performance bottleneck. Addressing this communication issue has become a prominent research topic. In this paper, we provide a comprehensive survey of the communication-efficient distributed training algorithms, focusing on both system-level and algorithmic-level optimizations. We first propose a taxonomy of data-parallel distributed training algorithms that incorporates four primary dimensions: communication synchronization, system architectures, compression techniques, and parallelism of communication and computing tasks. We then investigate state-of-the-art studies that address problems in these four dimensions. We also compare the convergence rates of different algorithms to understand their convergence speed. Additionally, we conduct extensive experiments to empirically compare the convergence performance of various mainstream distributed training algorithms. Based on our system-level communication cost analysis, theoretical and experimental convergence speed comparison, we provide readers with an understanding of which algorithms are more efficient under specific distributed environments. Our research also extrapolates potential directions for further optimizations.

翻译：分布式深度学习（DL）近年来因模型和数据集规模增大而广泛采用多计算设备（如GPU/TPU）以缩短训练时间。然而，系统可扩展性受限于通信成为性能瓶颈。解决通信问题已成为重要研究方向。本文对通信高效的分布式训练算法进行了全面综述，重点涵盖系统级和算法级优化。我们首先提出一种数据并行分布式训练算法的分类体系，包含四个主要维度：通信同步、系统架构、压缩技术以及通信与计算任务的并行性。随后，我们调研了针对这四个维度问题的前沿研究，并比较不同算法的收敛速率以理解其收敛速度。此外，我们通过大量实验实证对比了多种主流分布式训练算法的收敛性能。基于系统级通信成本分析及理论与实验收敛速度对比，我们为读者提供了在特定分布式环境下更高效算法的选取依据。本研究还探讨了进一步优化的潜在方向。