Recent work has shown that methods like SAM which either explicitly or implicitly penalize second order information can improve generalization in deep learning. Seemingly similar methods like weight noise and gradient penalties often fail to provide such benefits. We show that these differences can be explained by the structure of the Hessian of the loss. First, we show that a common decomposition of the Hessian can be quantitatively interpreted as separating the feature exploitation from feature exploration. The feature exploration, which can be described by the Nonlinear Modeling Error matrix (NME), is commonly neglected in the literature since it vanishes at interpolation. Our work shows that the NME is in fact important as it can explain why gradient penalties are sensitive to the choice of activation function. Using this insight we design interventions to improve performance. We also provide evidence that challenges the long held equivalence of weight noise and gradient penalties. This equivalence relies on the assumption that the NME can be ignored, which we find does not hold for modern networks since they involve significant feature learning. We find that regularizing feature exploitation but not feature exploration yields performance similar to gradient penalties.

翻译：近期研究表明，SAM等显式或隐式惩罚二阶信息的方法能够提升深度学习中的泛化性能。然而，权值噪声与梯度惩罚等看似相似的方法却往往无法带来此类收益。我们证明，这些差异可通过损失函数海森矩阵的结构加以解释。首先，我们证明海森矩阵的常见分解可被定量解释为将特征利用与特征探索相分离。特征探索可通过非线性建模误差矩阵（NME）描述，因其在插值点处消失而常被文献所忽视。本研究揭示NME实则至关重要：它能解释梯度惩罚对激活函数选择的敏感性。基于这一洞见，我们设计了若干干预措施以提升性能。此外，我们提供了质疑权值噪声与梯度惩罚长期等价关系的证据。这一等价关系依赖于可忽略NME的假设，但本研究发现该假设对现代网络不成立——这些网络涉及显著的特征学习。我们进一步发现，仅对特征利用进行正则化而不涉及特征探索时，其性能表现与梯度惩罚相似。