Gradient sparsification is a widely adopted solution for reducing the excessive communication traffic in distributed deep learning. However, most existing gradient sparsifiers have relatively poor scalability because of considerable computational cost of gradient selection and/or increased communication traffic owing to gradient build-up. To address these challenges, we propose a novel gradient sparsification scheme, DEFT, that partitions the gradient selection task into sub tasks and distributes them to workers. DEFT differs from existing sparsifiers, wherein every worker selects gradients among all gradients. Consequently, the computational cost can be reduced as the number of workers increases. Moreover, gradient build-up can be eliminated because DEFT allows workers to select gradients in partitions that are non-intersecting (between workers). Therefore, even if the number of workers increases, the communication traffic can be maintained as per user requirement. To avoid the loss of significance of gradient selection, DEFT selects more gradients in the layers that have a larger gradient norm than the other layers. Because every layer has a different computational load, DEFT allocates layers to workers using a bin-packing algorithm to maintain a balanced load of gradient selection between workers. In our empirical evaluation, DEFT shows a significant improvement in training performance in terms of speed in gradient selection over existing sparsifiers while achieving high convergence performance.

翻译：梯度稀疏化是分布式深度学习中降低过高通信流量的广泛应用方案。然而，现有大多数梯度稀疏化方法因梯度选择的计算开销较大，或梯度累积导致的通信流量增加，其可扩展性较差。针对这些挑战，我们提出了一种新型梯度稀疏化方案DEFT，该方案将梯度选择任务拆分为子任务并分配给各工作节点。DEFT与现有稀疏化方法不同——后者中每个工作节点需在所有梯度中进行选择。因此，随着工作节点数量增加，计算开销得以降低。此外，由于DEFT允许工作节点在互不重叠的分区中选择梯度，梯度累积问题可被消除。即使工作节点数量增加，通信流量仍可按用户需求维持。为避免梯度选择有效性的损失，DEFT在梯度范数较大的层中选择更多梯度。由于各层计算负载不同，DEFT采用箱体打包算法将各层分配给工作节点，以维持工作节点间梯度选择的负载均衡。实验评估表明，DEFT在梯度选择速度方面较现有稀疏化方法显著提升了训练性能，同时实现了较高的收敛性能。