In this paper, we study the implicit regularization of stochastic gradient descent (SGD) through the lens of {\em dynamical stability} (Wu et al., 2018). We start by revising existing stability analyses of SGD, showing how the Frobenius norm and trace of Hessian relate to different notions of stability. Notably, if a global minimum is linearly stable for SGD, then the trace of Hessian must be less than or equal to $2/\eta$, where $\eta$ denotes the learning rate. By contrast, for gradient descent (GD), the stability imposes a similar constraint but only on the largest eigenvalue of Hessian. We then turn to analyze the generalization properties of these stable minima, focusing specifically on two-layer ReLU networks and diagonal linear networks. Notably, we establish the {\em equivalence} between these metrics of sharpness and certain parameter norms for the two models, which allows us to show that the stable minima of SGD provably generalize well. By contrast, the stability-induced regularization of GD is provably too weak to ensure satisfactory generalization. This discrepancy provides an explanation of why SGD often generalizes better than GD. Note that the learning rate (LR) plays a pivotal role in the strength of stability-induced regularization. As the LR increases, the regularization effect becomes more pronounced, elucidating why SGD with a larger LR consistently demonstrates superior generalization capabilities. Additionally, numerical experiments are provided to support our theoretical findings.

翻译：摘要：本文通过动力学稳定性（Wu等人，2018）的视角研究随机梯度下降（SGD）的隐式正则化。我们首先修正现有的SGD稳定性分析，展示Hessian矩阵的Frobenius范数和迹如何与不同稳定性概念相关联。值得注意的是，若SGD的全局最小值具有线性稳定性，则Hessian矩阵的迹必须小于或等于$2/\eta$，其中$\eta$表示学习率。相比之下，梯度下降（GD）的稳定性虽施加了类似约束，但仅针对Hessian矩阵的最大特征值。随后，我们分析这些稳定最小值的泛化性质，特别关注双层ReLU网络和对角线性网络。值得注意的是，我们建立了这两个模型中锐度度量与特定参数范数之间的**等价性**，从而证明SGD的稳定最小值可理论保证良好泛化。相比之下，GD的稳定性诱导正则化理论上过弱，难以确保令人满意的泛化性能。这一差异解释了为何SGD通常比GD具有更好的泛化能力。需注意学习率（LR）在稳定性诱导正则化的强度中起关键作用。随着学习率增加，正则化效应愈发显著，这阐明了为何采用较大学习率的SGD始终展现出更优越的泛化能力。此外，数值实验验证了我们的理论发现。