Adaptive gradient methods have become popular in optimizing deep neural networks; recent examples include AdaGrad and Adam. Although Adam usually converges faster, variations of Adam, for instance, the AdaBelief algorithm, have been proposed to enhance Adam's poor generalization ability compared to the classical stochastic gradient method. This paper develops a generic framework for adaptive gradient methods that solve non-convex optimization problems. We first model the adaptive gradient methods in a state-space framework, which allows us to present simpler convergence proofs of adaptive optimizers such as AdaGrad, Adam, and AdaBelief. We then utilize the transfer function paradigm from classical control theory to propose a new variant of Adam, coined AdamSSM. We add an appropriate pole-zero pair in the transfer function from squared gradients to the second moment estimate. We prove the convergence of the proposed AdamSSM algorithm. Applications on benchmark machine learning tasks of image classification using CNN architectures and language modeling using LSTM architecture demonstrate that the AdamSSM algorithm improves the gap between generalization accuracy and faster convergence than the recent adaptive gradient methods.

翻译：自适应梯度方法在深度神经网络优化中已变得十分流行，近期实例包括AdaGrad和Adam。尽管Adam通常收敛更快，但为解决与经典随机梯度方法相比Adam泛化能力较差的不足，已提出多种Adam变体（例如AdaBelief算法）。本文为求解非凸优化问题的自适应梯度方法开发了一个通用框架。我们首先在状态空间框架中建模自适应梯度方法，这使得我们能够给出AdaGrad、Adam和AdaBelief等自适应优化器更简洁的收敛性证明。随后利用经典控制理论中的传递函数范式，提出Adam的新变体——AdamSSM。我们在从平方梯度到二阶矩估计的传递函数中引入适当的极点-零点对，证明了所提AdamSSM算法的收敛性。在基于CNN架构的图像分类与基于LSTM架构的语言建模等基准机器学习任务上的应用表明，相比近期自适应梯度方法，AdamSSM算法在泛化精度与更快收敛速度之间实现了更优的平衡。