We consider a realistic decentralized setup with bandwidth-constrained communication and derive optimal time complexities for non-convex stochastic parallel and asynchronous optimization (up to logarithmic factors). We develop the corresponding methods, Grace SGD and Leon SGD, for both homogeneous and heterogeneous settings. Unlike previous work, our optimal bounds are characterized in terms of min-cut/max-flow quantities and rely on tools from Gomory-Hu trees and Steiner Tree Packing problems, providing tighter and more practical complexities.

翻译：我们考虑一个具有带宽受限通信的现实分散式设置，并推导了非凸随机并行和异步优化（至对数因子）的最优时间复杂度。我们针对同质和异质设置分别开发了相应的方法：Grace SGD 和 Leon SGD。与以往工作不同，我们的最优界通过最小割/最大流量来表征，并依赖于 Gomory-Hu 树和斯坦纳树打包问题的工具，从而提供了更紧致且更具实用性的复杂度。