We propose SING (StabIlized and Normalized Gradient), a plug-and-play technique that improves the stability and generalization of the Adam(W) optimizer. SING is straightforward to implement and has minimal computational overhead, requiring only a layer-wise standardization of the gradients fed to Adam(W) without introducing additional hyper-parameters. We support the effectiveness and practicality of the proposed approach by showing improved results on a wide range of architectures, problems (such as image classification, depth estimation, and natural language processing), and in combination with other optimizers. We provide a theoretical analysis of the convergence of the method, and we show that by virtue of the standardization, SING can escape local minima narrower than a threshold that is inversely proportional to the network's depth.

翻译：我们提出SING（稳定化与归一化梯度），一种即插即用技术，可提升Adam(W)优化器的稳定性与泛化性能。SING实现简单且计算开销极小，仅需对输入Adam(W)的梯度进行逐层标准化处理，无需引入额外超参数。通过多种架构、问题场景（如图像分类、深度估计与自然语言处理）以及与其他优化器组合的改进结果，我们验证了该方法的有效性与实用性。我们提供了该方法收敛性的理论分析，并证明通过标准化机制，SING能够逃离窄于某一阈值的局部最小值，该阈值与网络深度成反比。