We study the implicit bias of Sharpness-Aware Minimization (SAM) when training $L$-layer linear diagonal networks on linearly separable binary classification. For linear models ($L=1$), both $\ell_\infty$- and $\ell_2$-SAM recover the $\ell_2$ max-margin classifier, matching gradient descent (GD). However, for depth $L = 2$, the behavior changes drastically -- even on a single-example dataset. For $\ell_\infty$-SAM, the limit direction depends critically on initialization and can converge to $\mathbf{0}$ or to any standard basis vector, in stark contrast to GD, whose limit aligns with the basis vector of the dominant data coordinate. For $\ell_2$-SAM, we show that although its limit direction matches the $\ell_1$ max-margin solution as in the case of GD, its finite-time dynamics exhibit a phenomenon we call "sequential feature amplification", in which the predictor initially relies on minor coordinates and gradually shifts to larger ones as training proceeds or initialization increases. Our theoretical analysis attributes this phenomenon to $\ell_2$-SAM's gradient normalization factor applied in its perturbation, which amplifies minor coordinates early and allows major ones to dominate later, giving a concrete example where infinite-time implicit-bias analyses are insufficient. Synthetic and real-data experiments corroborate our findings.

翻译：我们研究在训练$L$层线性对角网络进行线性可分二分类时，锐度感知最小化（SAM）的隐式偏差。对于线性模型（$L=1$），$\ell_\infty$-SAM和$\ell_2$-SAM均恢复$\ell_2$最大间隔分类器，与梯度下降（GD）一致。然而，当深度$L=2$时，行为发生剧烈变化——即使在单样本数据集上也是如此。对于$\ell_\infty$-SAM，极限方向关键依赖于初始化，可能收敛到$\mathbf{0}$或任意标准基向量，这与GD形成鲜明对比，GD的极限方向与主导数据坐标的基向量对齐。对于$\ell_2$-SAM，我们证明尽管其极限方向与GD情况下的$\ell_1$最大间隔解匹配，但其有限时间动力学展现出一种称为“顺序特征放大”的现象，即预测器最初依赖次要坐标，随着训练进行或初始值增大，逐渐转向较大坐标。我们的理论分析将该现象归因于$\ell_2$-SAM在其扰动中应用的梯度归一化因子，该因子在早期放大次要坐标，后期让主要坐标占主导，从而给出一个无限时间隐式偏差分析不足的具体实例。合成和真实数据实验佐证了我们的发现。