Traditional analyses of gradient descent show that when the largest eigenvalue of the Hessian, also known as the sharpness $S(\theta)$, is bounded by $2/\eta$, training is "stable" and the training loss decreases monotonically. Recent works, however, have observed that this assumption does not hold when training modern neural networks with full batch or large batch gradient descent. Most recently, Cohen et al. (2021) observed two important phenomena. The first, dubbed progressive sharpening, is that the sharpness steadily increases throughout training until it reaches the instability cutoff $2/\eta$. The second, dubbed edge of stability, is that the sharpness hovers at $2/\eta$ for the remainder of training while the loss continues decreasing, albeit non-monotonically. We demonstrate that, far from being chaotic, the dynamics of gradient descent at the edge of stability can be captured by a cubic Taylor expansion: as the iterates diverge in direction of the top eigenvector of the Hessian due to instability, the cubic term in the local Taylor expansion of the loss function causes the curvature to decrease until stability is restored. This property, which we call self-stabilization, is a general property of gradient descent and explains its behavior at the edge of stability. A key consequence of self-stabilization is that gradient descent at the edge of stability implicitly follows projected gradient descent (PGD) under the constraint $S(\theta) \le 2/\eta$. Our analysis provides precise predictions for the loss, sharpness, and deviation from the PGD trajectory throughout training, which we verify both empirically in a number of standard settings and theoretically under mild conditions. Our analysis uncovers the mechanism for gradient descent's implicit bias towards stability.

翻译：传统梯度下降分析表明，当Hessian矩阵的最大特征值（即锐度$S(\theta)$）被$2/\eta$界定时，训练是"稳定的"且训练损失单调递减。然而近期研究发现，在使用全批次或大批次梯度下降训练现代神经网络时，该假设并不成立。最近Cohen等人（2021）观察到两个重要现象：一是被称为渐进锐化的现象——锐度在训练过程中持续增加直至达到不稳定性阈值$2/\eta$；二是被称为稳定边缘的现象——锐度在剩余训练阶段维持在$2/\eta$附近，此时损失虽非单调但仍持续下降。我们证明，梯度下降在稳定边缘的动力学远非混沌无序，而可通过三次泰勒展开精确描述：当迭代因不稳定性沿Hessian最大特征向量方向发散时，损失函数局部泰勒展开中的三次项导致曲率下降，直至稳定性恢复。这种被称为自稳定的性质是梯度下降的普遍特性，并解释了其在稳定边缘的行为。自稳定的关键结果在于，稳定边缘的梯度下降隐式遵循投影梯度下降（PGD），其约束条件为$S(\theta) \le 2/\eta$。我们的分析为整个训练过程中的损失、锐度及与PGD轨迹的偏差提供了精确预测，并在多个标准设置下通过实证验证，同时在温和理论条件下得到证实。该分析揭示了梯度下降向稳定性隐式偏差的内在机制。