Large neural networks require enormous computational clusters of machines. Model-parallel training, when the model architecture is partitioned sequentially between workers, is a popular approach for training modern models. Information compression can be applied to decrease workers communication time, as it is often a bottleneck in such systems. This work explores how simultaneous compression of activations and gradients in model-parallel distributed training setup affects convergence. We analyze compression methods such as quantization and TopK compression, and also experiment with error compensation techniques. Moreover, we employ TopK with AQ-SGD per-batch error feedback approach. We conduct experiments on image classification and language model fine-tuning tasks. Our findings demonstrate that gradients require milder compression rates than activations. We observe that $K=10\%$ is the lowest TopK compression level, which does not harm model convergence severely. Experiments also show that models trained with TopK perform well only when compression is also applied during inference. We find that error feedback techniques do not improve model-parallel training compared to plain compression, but allow model inference without compression with almost no quality drop. Finally, when applied with the AQ-SGD approach, TopK stronger than with $ K=30\%$ worsens model performance significantly.

翻译：大型神经网络需要庞大的机器计算集群。模型并行训练（将模型架构按顺序划分到各工作节点）是训练现代模型的流行方法。通过信息压缩技术可减少工作节点间的通信时间——这通常是此类系统的性能瓶颈。本研究探讨模型并行分布式训练中激活值与梯度同步压缩对收敛性的影响。我们分析了量化、TopK压缩等方法，并实验了误差补偿技术。此外，我们采用带有AQ-SGD逐批次误差反馈机制的TopK方法。在图像分类与语言模型微调任务上开展实验。研究结果表明：梯度需要比激活值更温和的压缩率。我们观察到K=10%是TopK压缩的最低有效阈值，低于此值会严重损害模型收敛性。实验还显示，仅当推理阶段也使用压缩时，基于TopK训练的模型才能保持良好性能。我们发现与直接压缩相比，误差反馈技术并未改善模型并行训练效果，但能使模型在无压缩推理时几乎不损失精度。最终，当采用AQ-SGD方法时，压缩率超过K=30%的TopK会显著恶化模型性能。