Communication compression is a crucial technique for modern distributed learning systems to alleviate their communication bottlenecks over slower networks. Despite recent intensive studies of gradient compression for data parallel-style training, compressing the activations for models trained with pipeline parallelism is still an open problem. In this paper, we propose AC-SGD, a novel activation compression algorithm for communication-efficient pipeline parallelism training over slow networks. Different from previous efforts in activation compression, instead of compressing activation values directly, AC-SGD compresses the changes of the activations. This allows us to show, to the best of our knowledge for the first time, that one can still achieve $O(1/\sqrt{T})$ convergence rate for non-convex objectives under activation compression, without making assumptions on gradient unbiasedness that do not hold for deep learning models with non-linear activation functions.We then show that AC-SGD can be optimized and implemented efficiently, without additional end-to-end runtime overhead.We evaluated AC-SGD to fine-tune language models with up to 1.5 billion parameters, compressing activations to 2-4 bits.AC-SGD provides up to 4.3X end-to-end speed-up in slower networks, without sacrificing model quality. Moreover, we also show that AC-SGD can be combined with state-of-the-art gradient compression algorithms to enable "end-to-end communication compression: All communications between machines, including model gradients, forward activations, and backward gradients are compressed into lower precision.This provides up to 4.9X end-to-end speed-up, without sacrificing model quality.

翻译：通信压缩是现代分布式学习系统在慢速网络上缓解通信瓶颈的关键技术。尽管近年来关于数据并行训练中梯度压缩的研究密集开展，但针对流水线并行训练中模型激活值的压缩仍是一个开放性问题。本文提出AC-SGD——一种面向慢速网络上通信高效流水线并行训练的新型激活压缩算法。与以往直接压缩激活值的做法不同，AC-SGD通过压缩激活值的变化量来实现压缩。这使得我们能够首次证明：在激活压缩下，无需对深度学习中非线性激活函数模型无法满足的梯度无偏性假设，即可实现非凸目标函数$O(1/\sqrt{T})$的收敛速率。进一步，我们展示了AC-SGD可被优化并高效实现，且不增加端到端运行时开销。我们评估了AC-SGD在多达15亿参数语言模型上的微调效果，将激活值压缩至2-4比特。在慢速网络上，AC-SGD可在不牺牲模型质量的前提下实现高达4.3倍的端到端加速。此外，我们还证明AC-SGD可与最先进的梯度压缩算法结合，实现"端到端通信压缩"：机器间的所有通信——包括模型梯度、前向激活值和反向梯度——均被压缩至低精度。这在不牺牲模型质量的前提下提供了高达4.9倍的端到端加速。