In this paper, we study the implicit regularization of stochastic gradient descent (SGD) through the lens of {\em dynamical stability} (Wu et al., 2018). We start by revising existing stability analyses of SGD, showing how the Frobenius norm and trace of Hessian relate to different notions of stability. Notably, if a global minimum is linearly stable for SGD, then the trace of Hessian must be less than or equal to $2/\eta$, where $\eta$ denotes the learning rate. By contrast, for gradient descent (GD), the stability imposes a similar constraint but only on the largest eigenvalue of Hessian. We then turn to analyze the generalization properties of these stable minima, focusing specifically on two-layer ReLU networks and diagonal linear networks. Notably, we establish the {\em equivalence} between these metrics of sharpness and certain parameter norms for the two models, which allows us to show that the stable minima of SGD provably generalize well. By contrast, the stability-induced regularization of GD is provably too weak to ensure satisfactory generalization. This discrepancy provides an explanation of why SGD often generalizes better than GD. Note that the learning rate (LR) plays a pivotal role in the strength of stability-induced regularization. As the LR increases, the regularization effect becomes more pronounced, elucidating why SGD with a larger LR consistently demonstrates superior generalization capabilities. Additionally, numerical experiments are provided to support our theoretical findings.

翻译：在本文中，我们通过动态稳定性（Wu等人，2018）的视角研究随机梯度下降（SGD）的隐式正则化。首先，我们重新审视现有关于SGD稳定性的分析，展示Hessian矩阵的Frobenius范数与迹如何与不同的稳定性概念相关联。值得注意的是，如果全局最小值在SGD下是线性稳定的，则Hessian矩阵的迹必须小于或等于$2/\eta$，其中$\eta$表示学习率。相比之下，对于梯度下降（GD），稳定性施加了类似的约束，但仅针对Hessian矩阵的最大特征值。随后，我们转向分析这些稳定最小值的泛化性质，特别关注两层ReLU网络和对角线性网络。值得注意的是，我们建立了这两个模型中锐度度量与特定参数范数之间的等价性，从而得以证明SGD的稳定最小值可被证明具有良好的泛化性能。相比之下，GD由稳定性诱导的正则化被证明过于微弱，无法保证令人满意的泛化性能。这一差异解释了为何SGD通常比GD泛化性能更好。需指出的是，学习率（LR）在稳定性诱导的正则化强度中起着关键作用。随着学习率增大，正则化效果更为显著，这也阐释了为何采用较大学习率的SGD始终展现出更优的泛化能力。此外，我们通过数值实验支持了理论发现。