This paper studies the algorithmic stability and generalizability of decentralized stochastic gradient descent (D-SGD). We prove that the consensus model learned by D-SGD is $\mathcal{O}{(N^{-1}+m^{-1} +\lambda^2)}$-stable in expectation in the non-convex non-smooth setting, where $N$ is the total sample size, $m$ is the worker number, and $1+\lambda$ is the spectral gap that measures the connectivity of the communication topology. These results then deliver an $\mathcal{O}{(N^{-(1+\alpha)/2}+ m^{-(1+\alpha)/2}+\lambda^{1+\alpha} + \phi_{\mathcal{S}})}$ in-average generalization bound, which is non-vacuous even when $\lambda$ is closed to $1$, in contrast to vacuous as suggested by existing literature on the projected version of D-SGD. Our theory indicates that the generalizability of D-SGD is positively correlated with the spectral gap, and can explain why consensus control in initial training phase can ensure better generalization. Experiments of VGG-11 and ResNet-18 on CIFAR-10, CIFAR-100 and Tiny-ImageNet justify our theory. To our best knowledge, this is the first work on the topology-aware generalization of vanilla D-SGD. Code is available at https://github.com/Raiden-Zhu/Generalization-of-DSGD.

翻译：本文研究分散式随机梯度下降（D-SGD）的算法稳定性与泛化能力。我们证明在非凸非光滑设定下，D-SGD习得的共识模型在期望意义上是 $\mathcal{O}{(N^{-1}+m^{-1} +\lambda^2)}$-稳定的，其中 $N$ 为总样本量，$m$ 为工作者数量，$1+\lambda$ 为衡量通信拓扑连通性的谱间隙。这些结果进一步推导出 $\mathcal{O}{(N^{-(1+\alpha)/2}+ m^{-(1+\alpha)/2}+\lambda^{1+\alpha} + \phi_{\mathcal{S}})}$ 的平均泛化界，即使当 $\lambda$ 接近1时该界仍非平凡——这与现有文献关于投影版D-SGD的平凡结论形成对比。我们的理论表明D-SGD的泛化能力与谱间隙正相关，并可解释为何初始训练阶段的共识控制能确保更优泛化性能。基于VGG-11和ResNet-18在CIFAR-10、CIFAR-100和Tiny-ImageNet上的实验验证了我们的理论。据我们所知，这是首个针对原始版D-SGD的拓扑感知泛化工作。代码开源于 https://github.com/Raiden-Zhu/Generalization-of-DSGD。