Communication compression is an essential strategy for alleviating communication overhead by reducing the volume of information exchanged between computing nodes in large-scale distributed stochastic optimization. Although numerous algorithms with convergence guarantees have been obtained, the optimal performance limit under communication compression remains unclear. In this paper, we investigate the performance limit of distributed stochastic optimization algorithms employing communication compression. We focus on two main types of compressors, unbiased and contractive, and address the best-possible convergence rates one can obtain with these compressors. We establish the lower bounds for the convergence rates of distributed stochastic optimization in six different settings, combining strongly-convex, generally-convex, or non-convex functions with unbiased or contractive compressor types. To bridge the gap between lower bounds and existing algorithms' rates, we propose NEOLITHIC, a nearly optimal algorithm with compression that achieves the established lower bounds up to logarithmic factors under mild conditions. Extensive experimental results support our theoretical findings. This work provides insights into the theoretical limitations of existing compressors and motivates further research into fundamentally new compressor properties.

翻译：通信压缩是通过减少大规模分布式随机优化中计算节点间交换信息量来缓解通信开销的关键策略。尽管已有众多具备收敛保证的算法，但通信压缩下的最优性能极限仍不明确。本文研究采用通信压缩的分布式随机优化算法的性能极限。我们聚焦于无偏压缩器和收缩型压缩器这两类主要压缩器，探讨利用这些压缩器所能实现的最优收敛速率。针对强凸、一般凸或非凸函数与无偏或收缩型压缩器类型的六种不同组合，我们建立了分布式随机优化收敛速率的下界。为弥合下界与现有算法速率之间的差距，我们提出NEOLITHIC——一种近乎最优的压缩算法，在温和条件下可达所建立下界（仅相差对数因子）。大量实验结果支持了我们的理论发现。本研究揭示了现有压缩器的理论局限性，并推动对全新压缩器特性的深入探索。