Data-parallel SGD is the de facto algorithm for distributed optimization, especially for large scale machine learning. Despite its merits, communication bottleneck is one of its persistent issues. Most compression schemes to alleviate this either assume noiseless communication links, or fail to achieve good performance on practical tasks. In this paper, we close this gap and introduce LASER: LineAr CompreSsion in WirEless DistRibuted Optimization. LASER capitalizes on the inherent low-rank structure of gradients and transmits them efficiently over the noisy channels. Whilst enjoying theoretical guarantees similar to those of the classical SGD, LASER shows consistent gains over baselines on a variety of practical benchmarks. In particular, it outperforms the state-of-the-art compression schemes on challenging computer vision and GPT language modeling tasks. On the latter, we obtain $50$-$64 \%$ improvement in perplexity over our baselines for noisy channels.

翻译：数据并行SGD是分布式优化的事实标准算法，尤其适用于大规模机器学习。尽管其优势显著，通信瓶颈仍是持续存在的问题。大多数缓解此问题的压缩方案要么假设无噪声通信链路，要么无法在实际任务中取得良好性能。本文填补了这一空白，提出LASER：无线分布式优化中的线性压缩（LineAr CompreSsion in WirEless DistRibuted Optimization）。LASER利用梯度固有的低秩结构，通过噪声信道高效传输梯度。在享有与经典SGD相似的理论保证的同时，LASER在多种实际基准测试中始终优于基线方法。尤其在具有挑战性的计算机视觉和GPT语言建模任务上，LASER超越了现有最先进的压缩方案。针对噪声信道下的GPT语言建模任务，相较于基线方法，我们获得的困惑度提升幅度为50%-64%。