Classical machine learning models such as deep neural networks are usually trained by using Stochastic Gradient Descent-based (SGD) algorithms. The classical SGD can be interpreted as a discretization of the stochastic gradient flow. In this paper we propose a novel, robust and accelerated stochastic optimizer that relies on two key elements: (1) an accelerated Nesterov-like Stochastic Differential Equation (SDE) and (2) its semi-implicit Gauss-Seidel type discretization. The convergence and stability of the obtained method, referred to as NAG-GS, are first studied extensively in the case of the minimization of a quadratic function. This analysis allows us to come up with an optimal learning rate in terms of the convergence rate while ensuring the stability of NAG-GS. This is achieved by the careful analysis of the spectral radius of the iteration matrix and the covariance matrix at stationarity with respect to all hyperparameters of our method. Further, we show that NAG- GS is competitive with state-of-the-art methods such as momentum SGD with weight decay and AdamW for the training of machine learning models such as the logistic regression model, the residual networks models on standard computer vision datasets, Transformers in the frame of the GLUE benchmark and the recent Vision Transformers.

翻译：经典机器学习模型（如深度神经网络）通常采用基于随机梯度下降（SGD）的算法进行训练。经典SGD可视为随机梯度流的离散化形式。本文提出一种新颖、鲁棒且加速的随机优化器，其核心依赖两个关键要素：（1）加速型Nesterov类随机微分方程（SDE）；（2）其半隐式Gauss-Seidel型离散化方法。该优化器被称为NAG-GS，我们首先针对二次函数最小化问题对其收敛性与稳定性进行深入分析。通过精细分析迭代矩阵谱半径以及稳态下所有超参数相关的协方差矩阵，我们推导出能在保证NAG-GS稳定性的同时实现最优收敛速率的学习率。进一步研究表明，NAG-GS在逻辑回归模型训练、标准计算机视觉数据集上的残差网络模型、GLUE基准框架下的Transformer模型以及近期视觉Transformer（Vision Transformer）等机器学习模型训练任务中，与动量SGD（含权重衰减）、AdamW等最先进方法具有同等竞争力。