Communication bottlenecks severely hinder the scalability of distributed neural network training, particularly in high-performance computing (HPC) environments. We introduce AB-training, a novel data-parallel method that leverages low-rank representations and independent training groups to significantly reduce communication overhead. Our experiments demonstrate an average reduction in network traffic of approximately 70.31\% across various scaling scenarios, increasing the training potential of communication-constrained systems and accelerating convergence at scale. AB-training also exhibits a pronounced regularization effect at smaller scales, leading to improved generalization while maintaining or even reducing training time. We achieve a remarkable 44.14 : 1 compression ratio on VGG16 trained on CIFAR-10 with minimal accuracy loss, and outperform traditional data parallel training by 1.55\% on ResNet-50 trained on ImageNet-2012. While AB-training is promising, our findings also reveal that large batch effects persist even in low-rank regimes, underscoring the need for further research into optimized update mechanisms for massively distributed training.

翻译：通信瓶颈严重制约了分布式神经网络训练的可扩展性，尤其在高性能计算环境中。本文提出AB-训练——一种新颖的数据并行方法，该方法利用低秩表示与独立训练组显著降低通信开销。实验表明，在不同扩展场景下网络流量平均降低约70.31%，有效提升了通信受限系统的训练潜力并加速了大规模收敛。AB-训练在较小规模下还表现出显著的正则化效应，在保持甚至缩短训练时间的同时提升了泛化性能。在CIFAR-10数据集训练的VGG16上实现了44.14:1的压缩比且精度损失极小，在ImageNet-2012数据集训练的ResNet-50上性能超越传统数据并行训练1.55%。尽管AB-训练前景广阔，我们的研究同时揭示了大批量效应在低秩体系中依然存在，这凸显了对大规模分布式训练优化更新机制进行深入研究的必要性。