We consider minimizing a perturbed function $F(W) = \mathbb{E}_{U}[f(W + U)]$, given a function $f: \mathbb{R}^d \rightarrow \mathbb{R}$ and a random sample $U$ from a distribution $\mathcal{P}$ with mean zero. When $\mathcal{P}$ is the isotropic Gaussian, $F(W)$ is roughly equal to $f(W)$ plus a penalty on the trace of $\nabla^2 f(W)$, scaled by the variance of $\mathcal{P}$. This penalty on the Hessian has the benefit of improving generalization, through PAC-Bayes analysis. It is useful in low-sample regimes, for instance, when a (large) pre-trained model is fine-tuned on a small data set. One way to minimize $F$ is by adding $U$ to $W$, and then run SGD. We observe, empirically, that this noise injection does not provide significant gains over SGD, in our experiments of conducting fine-tuning on three image classification data sets. We design a simple, practical algorithm that adds noise along both $U$ and $-U$, with the option of adding several perturbations and taking their average. We analyze the convergence of this algorithm, showing tight rates on the norm of the output's gradient. We provide a comprehensive empirical analysis of our algorithm, by first showing that in an over-parameterized matrix sensing problem, it can find solutions with lower test loss than naive noise injection. Then, we compare our algorithm with four sharpness-reducing training methods (such as the Sharpness-Aware Minimization (Foret et al., 2021)). We find that our algorithm can outperform them by up to 1.8% test accuracy, for fine-tuning ResNet on six image classification data sets. It leads to a 17.7% (and 12.8%) reduction in the trace (and largest eigenvalue) of the Hessian matrix of the loss surface. This form of regularization on the Hessian is compatible with $\ell_2$ weight decay (and data augmentation), in the sense that combining both can lead to improved empirical performance.

翻译：我们考虑最小化扰动函数 $F(W) = \mathbb{E}_{U}[f(W + U)]$，其中给定函数 $f: \mathbb{R}^d \rightarrow \mathbb{R}$ 以及来自均值为零的分布 $\mathcal{P}$ 的随机样本 $U$。当 $\mathcal{P}$ 为各向同性高斯分布时，$F(W)$ 近似等于 $f(W)$ 加上一个按 $\mathcal{P}$ 方差缩放的海森矩阵迹的惩罚项。这种对海森矩阵的惩罚通过 PAC-Bayes 分析具有提升泛化能力的优势，尤其适用于小样本场景，例如将（大型）预训练模型在小型数据集上进行微调。最小化 $F$ 的一种方法是在 $W$ 上添加 $U$ 后运行 SGD。我们通过实验观察到，在对三个图像分类数据集进行微调的过程中，这种噪声注入并未比标准 SGD 带来显著性能提升。为此，我们设计了一种简单实用的算法，该算法沿 $U$ 和 $-U$ 两个方向同时添加噪声，并支持添加多个扰动后取均值。我们分析了该算法的收敛性，给出了输出梯度范数的紧速率界。通过全面的实证分析，我们首先证明在过参数化的矩阵感知问题中，该算法能找到比朴素噪声注入测试损失更低的解。随后，我们将该算法与四种锐度降低训练方法（如锐度感知最小化（Foret 等人，2021））进行比较。实验表明，在六种图像分类数据集上对 ResNet 进行微调时，我们的算法可将其测试准确率提升高达 1.8%。该算法使损失曲面的海森矩阵迹（及最大特征值）分别降低 17.7%（和 12.8%）。这种海森矩阵正则化形式与 $\ell_2$ 权重衰减（及数据增强）兼容，即两者结合可进一步提升经验性能。