Adaptive optimizers, such as Adam, have achieved remarkable success in deep learning. A key component of these optimizers is the so-called preconditioning matrix, providing enhanced gradient information and regulating the step size of each gradient direction. In this paper, we propose a novel approach to designing the preconditioning matrix by utilizing the gradient difference between two successive steps as the diagonal elements. These diagonal elements are closely related to the Hessian and can be perceived as an approximation of the inner product between the Hessian row vectors and difference of the adjacent parameter vectors. Additionally, we introduce an auto-switching function that enables the preconditioning matrix to switch dynamically between Stochastic Gradient Descent (SGD) and the adaptive optimizer. Based on these two techniques, we develop a new optimizer named AGD that enhances the generalization performance. We evaluate AGD on public datasets of Natural Language Processing (NLP), Computer Vision (CV), and Recommendation Systems (RecSys). Our experimental results demonstrate that AGD outperforms the state-of-the-art (SOTA) optimizers, achieving highly competitive or significantly better predictive performance. Furthermore, we analyze how AGD is able to switch automatically between SGD and the adaptive optimizer and its actual effects on various scenarios. The code is available at https://github.com/intelligent-machine-learning/atorch/tree/main/atorch/optimizers.

翻译：自适应优化器（如Adam）在深度学习领域取得了显著成功。这些优化器的关键组件是所谓的预条件矩阵，它提供增强的梯度信息并调节每个梯度方向的步长。本文提出一种设计预条件矩阵的新方法，该方法利用连续两步间的梯度差作为对角元素。这些对角元素与Hessian矩阵密切相关，可视为Hessian行向量与相邻参数向量差之间内积的近似。此外，我们引入一种自动切换函数，使预条件矩阵能够在随机梯度下降（SGD）与自适应优化器之间动态切换。基于这两项技术，我们开发了一种名为AGD的新型优化器，其增强了泛化性能。我们在自然语言处理（NLP）、计算机视觉（CV）和推荐系统（RecSys）的公共数据集上评估AGD。实验结果表明，AGD优于当前最先进的优化器，取得了极具竞争力或显著更优的预测性能。此外，我们分析了AGD如何实现SGD与自适应优化器间的自动切换及其在不同场景中的实际效果。代码发布于https://github.com/intelligent-machine-learning/atorch/tree/main/atorch/optimizers。