Intensive communication and synchronization cost for gradients and parameters is the well-known bottleneck of distributed deep learning training. Based on the observations that Synchronous SGD (SSGD) obtains good convergence accuracy while asynchronous SGD (ASGD) delivers a faster raw training speed, we propose Several Steps Delay SGD (DeSGD) to combine their merits, aiming at tackling the communication bottleneck via communication sparsification. DeSGD explores both global synchronous updates in the parameter servers and asynchronous local updates in the workers in each periodic iteration. The periodic and flexible synchronization makes DeSGD achieve good convergence accuracy and fast training speed. To the best of our knowledge, we strike the new balance between synchronization quality and communication sparsification, and improve the trade-off between accuracy and training speed. Specifically, the core components of DeSGD include proper warm-up stage, steps delay stage, and our novel algorithm of global gradient for local update (GGLU). GGLU is critical for local update operations to effectively compensate the delayed local weights. Furthermore, we implement DeSGD on MXNet framework and comprehensively evaluate its performance with CIFAR-10 and ImageNet datasets. Experimental results show that DeSGD can accelerate distributed training speed under different experimental configurations, by up to 110%, while achieving good convergence accuracy.

翻译：基于同步 SGD (SSGD) 获得良好趋同准确性和快速培训速度的观测结果,我们提议采用若干步骤延迟 SGD (DeSGD) 来综合其优点, 目的是通过通信封闭解决通信瓶颈问题。 DeSGD 探索参数服务器全球同步更新和每期定期循环中工人不同步的本地更新。 定期和灵活同步使DeSGD 实现良好的趋同准确性和快速培训速度。 根据我们的知识,我们在同步质量和通信松散之间达成新的平衡,改进准确性和培训速度之间的交易。具体地说,DeSGD的核心组成部分包括适当的暖化阶段、步骤延迟阶段和我们全球梯度新奇的本地更新算法(GGLU )。 GGLU对于本地更新业务以有效补偿延迟的本地重量至关重要。 此外,我们根据我们的知识,在同步质量和通信松散之间实现新的平衡,同时通过不同图像模型的快速化模型,在IMX上展示其快速化的模型化模型,通过不同的模型显示其快速化模型框架和快速化的模型,通过不同的模型显示其快速化的模型化模型化的模型和加速化的模型化数据。