The recently proposed optimization algorithm for deep neural networks Sharpness Aware Minimization (SAM) suggests perturbing parameters before gradient calculation by a gradient ascent step to guide the optimization into parameter space regions of flat loss. While significant generalization improvements and thus reduction of overfitting could be demonstrated, the computational costs are doubled due to the additionally needed gradient calculation, making SAM unfeasible in case of limited computationally capacities. Motivated by Nesterov Accelerated Gradient (NAG) we propose Momentum-SAM (MSAM), which perturbs parameters in the direction of the accumulated momentum vector to achieve low sharpness without significant computational overhead or memory demands over SGD or Adam. We evaluate MSAM in detail and reveal insights on separable mechanisms of NAG, SAM and MSAM regarding training optimization and generalization. Code is available at https://github.com/MarlonBecker/MSAM.

翻译：近期提出的深度神经网络优化算法锐度感知最小化（SAM）建议在梯度计算前通过梯度上升步骤扰动参数，以引导优化进入损失平坦的参数空间区域。虽然该方法能显著提升泛化性能并减少过拟合，但由于需要额外梯度计算，计算成本翻倍，使得SAM在计算资源有限的情况下难以应用。受内斯特罗夫加速梯度（NAG）启发，我们提出动量-SAM（MSAM），该方法沿累积动量向量方向扰动参数，从而在不显著增加计算开销或内存需求（相较于SGD或Adam）的情况下实现低锐度。我们详细评估了MSAM，并揭示了NAG、SAM与MSAM在训练优化和泛化机制上的可分离特性。代码已开源至https://github.com/MarlonBecker/MSAM。