Recent experiments have shown that, often, when training a neural network with gradient descent (GD) with a step size $\eta$, the operator norm of the Hessian of the loss grows until it approximately reaches $2/\eta$, after which it fluctuates around this value. The quantity $2/\eta$ has been called the "edge of stability" based on consideration of a local quadratic approximation of the loss. We perform a similar calculation to arrive at an "edge of stability" for Sharpness-Aware Minimization (SAM), a variant of GD which has been shown to improve its generalization. Unlike the case for GD, the resulting SAM-edge depends on the norm of the gradient. Using three deep learning training tasks, we see empirically that SAM operates on the edge of stability identified by this analysis.

翻译：近期实验表明，当使用步长为 $\eta$ 的梯度下降法训练神经网络时，损失函数海森矩阵的谱范数通常增长至接近 $2/\eta$，随后围绕该值波动。基于损失函数的局部二次近似分析，$2/\eta$ 被定义为"稳定性边缘"。本文通过类似计算，为被证实能提升泛化性能的梯度下降变体——锐度感知最小化方法——推导出对应的"稳定性边缘"。与梯度下降不同，所得 SAM-稳定性边缘取决于梯度范数。通过三项深度学习训练任务，我们实证观察到锐度感知最小化方法在该分析所识别的稳定性边缘上运行。