This paper studies decentralized stochastic nonconvex optimization problem over row-stochastic networks. We consider the heavy-tailed gradient noise which is empirically observed in many popular real-world applications. Specifically, we propose a decentralized normalized stochastic gradient descent with Pull-Diag gradient tracking, which achieves approximate stationary points with the optimal sample complexity and the near-optimal communication complexity. We further follow our framework to study the setting of undirected networks, also achieving the nearly tight upper complexity bounds. Moreover, we conduct empirical studies to show the practical superiority of the proposed methods.

翻译：本文研究基于行随机网络的去中心化随机非凸优化问题。我们考虑了在众多实际应用中经验观测到的重尾梯度噪声。具体而言，我们提出了一种采用Pull-Diag梯度跟踪的去中心化归一化随机梯度下降方法，该方法能以最优样本复杂度和近最优通信复杂度达到近似驻点。我们进一步遵循该框架研究了无向网络设置，同样实现了近乎紧致的上界复杂度。此外，我们通过实证研究验证了所提出方法的实际优越性。