Large neural networks require enormous computational clusters of machines. Model-parallel training, when the model architecture is partitioned sequentially between workers, is a popular approach for training modern models. Information compression can be applied to decrease workers communication time, as it is often a bottleneck in such systems. This work explores how simultaneous compression of activations and gradients in model-parallel distributed training setup affects convergence. We analyze compression methods such as quantization and TopK compression, and also experiment with error compensation techniques. Moreover, we employ TopK with AQ-SGD per-batch error feedback approach. We conduct experiments on image classification and language model fine-tuning tasks. Our findings demonstrate that gradients require milder compression rates than activations. We observe that $K=10\%$ is the lowest TopK compression level, which does not harm model convergence severely. Experiments also show that models trained with TopK perform well only when compression is also applied during inference. We find that error feedback techniques do not improve model-parallel training compared to plain compression, but allow model inference without compression with almost no quality drop. Finally, when applied with the AQ-SGD approach, TopK stronger than with $ K=30\%$ worsens model performance significantly.

翻译：大型神经网络需要庞大的计算集群。当模型架构在计算节点间按顺序划分时，模型并行训练是训练现代模型的常用方法。由于通信时间往往成为此类系统的瓶颈，信息压缩技术可用于减少计算节点间的通信时间。本研究探讨了在模型并行分布式训练场景下，对激活值与梯度同时进行压缩如何影响收敛性。我们分析了量化和TopK压缩等方法，并尝试了误差补偿技术。此外，我们将TopK与基于每批次误差反馈的AQ-SGD方法结合使用。我们在图像分类和语言模型微调任务上进行了实验。结果表明，梯度所需的压缩率需低于激活值。我们观察到$K=10\%$是TopK压缩的最低阈值，在此阈值下模型收敛未受到严重损害。实验还表明，采用TopK训练的模型仅在推理阶段也应用压缩时才能取得良好效果。我们发现，与单纯压缩相比，误差反馈技术并未改善模型并行训练效果，但允许模型在无压缩推理时质量几乎不下降。最后，当结合AQ-SGD方法使用时，压缩率强于$K=30\%$的TopK会显著降低模型性能。