Error accumulation is effective for gradient sparsification in distributed settings: initially-unselected gradient entries are eventually selected as their accumulated error exceeds a certain level. The accumulation essentially behaves as a scaling of the learning rate for the selected entries. Although this property prevents the slow-down of lateral movements in distributed gradient descent, it can deteriorate convergence in some settings. This work proposes a novel sparsification scheme that controls the learning rate scaling of error accumulation. The development of this scheme follows two major steps: first, gradient sparsification is formulated as an inverse probability (inference) problem, and the Bayesian optimal sparsification mask is derived as a maximum-a-posteriori estimator. Using the prior distribution inherited from Top-k, we derive a new sparsification algorithm which can be interpreted as a regularized form of Top-k. We call this algorithm regularized Top-k (RegTop-k). It utilizes past aggregated gradients to evaluate posterior statistics of the next aggregation. It then prioritizes the local accumulated gradient entries based on these posterior statistics. We validate our derivation through various numerical experiments. In distributed linear regression, it is observed that while Top-k remains at a fixed distance from the global optimum, RegTop-k converges to the global optimum at significantly higher compression ratios. We further demonstrate the generalization of this observation by employing RegTop-k in distributed training of ResNet-18 on CIFAR-10, as well as fine-tuning of multiple computer vision models on the ImageNette dataset. Our numerical results confirm that as the compression ratio increases, RegTop-k sparsification noticeably outperforms Top-k.

翻译：在分布式环境中，误差累积对于梯度稀疏化是有效的：最初未被选中的梯度条目最终会被选中，因为其累积误差超过了特定阈值。这种累积本质上表现为对所选条目学习率的一种缩放。尽管这一特性防止了分布式梯度下降中横向移动的减缓，但在某些场景下可能恶化收敛性。本研究提出了一种新颖的稀疏化方案，用于控制误差累积带来的学习率缩放效应。该方案的开发遵循两个主要步骤：首先，将梯度稀疏化表述为一个逆概率（推断）问题，并推导出贝叶斯最优稀疏化掩码作为最大后验估计器。通过继承Top-k的先验分布，我们推导出一种新的稀疏化算法，可解释为Top-k的正则化形式。我们称此算法为正则化Top-k（RegTop-k）。它利用过去聚合的梯度来评估下一次聚合的后验统计量，并基于这些后验统计量对本地累积梯度条目进行优先级排序。我们通过多种数值实验验证了推导结果。在分布式线性回归中，观察到Top-k始终与全局最优解保持固定距离，而RegTop-k在显著更高的压缩比下收敛至全局最优解。我们进一步通过在CIFAR-10数据集上分布式训练ResNet-18，以及在ImageNette数据集上微调多个计算机视觉模型，验证了这一观测结果的普适性。数值结果证实，随着压缩比的提高，RegTop-k稀疏化方法明显优于Top-k。