We consider nonconvex stochastic optimization problems in the asynchronous centralized distributed setup where the communication times from workers to a server can not be ignored, and the computation and communication times are potentially different for all workers. Using an unbiassed compression technique, we develop a new method-Shadowheart SGD-that provably improves the time complexities of all previous centralized methods. Moreover, we show that the time complexity of Shadowheart SGD is optimal in the family of centralized methods with compressed communication. We also consider the bidirectional setup, where broadcasting from the server to the workers is non-negligible, and develop a corresponding method.

翻译：我们考虑异步中心化分布式环境下的非凸随机优化问题，其中从工作节点到服务器的通信时间不可忽略，且各工作节点的计算与通信时间可能互不相同。通过采用无偏压缩技术，我们提出一种新方法——Shadowheart SGD——该方法在理论上改进了所有现有中心化方法的时间复杂度。此外，我们证明在压缩通信的中心化方法族中，Shadowheart SGD的时间复杂度达到最优。同时，我们还考虑了服务器向工作节点广播不可忽略的双向通信场景，并开发了相应的方法。