Recent experiments have shown that, often, when training a neural network with gradient descent (GD) with a step size $\eta$, the operator norm of the Hessian of the loss grows until it approximately reaches $2/\eta$, after which it fluctuates around this value. The quantity $2/\eta$ has been called the "edge of stability" based on consideration of a local quadratic approximation of the loss. We perform a similar calculation to arrive at an "edge of stability" for Sharpness-Aware Minimization (SAM), a variant of GD which has been shown to improve its generalization. Unlike the case for GD, the resulting SAM-edge depends on the norm of the gradient. Using three deep learning training tasks, we see empirically that SAM operates on the edge of stability identified by this analysis.

翻译：近期实验表明，当使用步长$\eta$的梯度下降法训练神经网络时，损失函数Hessian矩阵的算子范数会逐渐增长，直至接近$2/\eta$，随后在该值附近波动。基于损失函数的局部二次近似，$2/\eta$被称为"稳定边界"。我们通过类似计算得出了锐度感知最小化（SAM）的"稳定边界"——这是一种已被证明能提升泛化性能的梯度下降变体。与梯度下降法不同，SAM的稳定边界取决于梯度范数。通过在三个深度学习训练任务中的实证研究，我们观察到SAM确实在本文分析所识别的稳定边界上运行。