Recent findings by Cohen et al., 2021, demonstrate that when training neural networks using full-batch gradient descent with a step size of $η$, the largest eigenvalue $λ_{\max}$ of the full-batch Hessian consistently stabilizes around $2/η$. These results have significant implications for convergence and generalization. This, however, is not the case for mini-batch optimization algorithms, limiting the broader applicabilityof the consequences of these findings. We show mini-batch Stochastic Gradient Descent (SGD) trains in a different regime we term Edge of Stochastic Stability (EoSS). In this regime, what stabilizes at $2/η$ is Batch Sharpness: the expected directional curvature of mini-batch Hessians along their corresponding stochastic gradients. As a consequence $λ_{\max}$ -- which is generally smaller than Batch Sharpness -- is suppressed, aligning with the long-standing empirical observation that smaller batches and larger step sizes favor flatter minima. We further discuss implications for mathematical modeling of SGD trajectories.

翻译：Cohen等人（2021年）的最新研究结果表明，在使用步长为$η$的全批次梯度下降训练神经网络时，全批次Hessian矩阵的最大特征值$λ_{\max}$会持续稳定在$2/η$附近。这些发现对收敛性和泛化性具有重要影响。然而，对于小批次优化算法而言，情况并非如此，这限制了上述研究结论的广泛适用性。我们证明小批次随机梯度下降（SGD）在一个我们称为随机稳定性边缘（EoSS）的不同机制中进行训练。在此机制中，稳定在$2/η$的是批次锐度：即小批次Hessian矩阵沿其对应随机梯度方向的期望曲率。因此，通常小于批次锐度的$λ_{\max}$会受到抑制，这与长期以来的经验观察结果一致：较小的批次和较大的步长有利于找到更平坦的极小值。我们进一步讨论了这一发现对SGD轨迹数学建模的影响。