Convolutional neural networks (CNNs) are trained using stochastic gradient descent (SGD)-based optimizers. Recently, the adaptive moment estimation (Adam) optimizer has become very popular due to its adaptive momentum, which tackles the dying gradient problem of SGD. Nevertheless, existing optimizers are still unable to exploit the optimization curvature information efficiently. This paper proposes a new AngularGrad optimizer that considers the behavior of the direction/angle of consecutive gradients. This is the first attempt in the literature to exploit the gradient angular information apart from its magnitude. The proposed AngularGrad generates a score to control the step size based on the gradient angular information of previous iterations. Thus, the optimization steps become smoother as a more accurate step size of immediate past gradients is captured through the angular information. Two variants of AngularGrad are developed based on the use of Tangent or Cosine functions for computing the gradient angular information. Theoretically, AngularGrad exhibits the same regret bound as Adam for convergence purposes. Nevertheless, extensive experiments conducted on benchmark data sets against state-of-the-art methods reveal a superior performance of AngularGrad. The source code will be made publicly available at: https://github.com/mhaut/AngularGrad.

翻译：卷积神经网络（CNN）通常采用基于随机梯度下降（SGD）的优化器进行训练。近年来，自适应矩估计（Adam）优化器因其自适应动量机制而广受欢迎，有效缓解了SGD的梯度消亡问题。然而，现有优化器仍难以高效利用优化曲率信息。本文提出了一种新型AngularGrad优化器，该优化器考虑了连续梯度方向/角度的行为特征。这是学界首次尝试在梯度幅度之外挖掘其角度信息。所提出的AngularGrad通过利用先前迭代梯度的角度信息生成一个评分，用于控制步长。由于通过角度信息捕捉到了最近历史梯度的更精确步长，优化过程因此变得更加平滑。基于正切函数或余弦函数计算梯度角度信息，我们分别开发了AngularGrad的两种变体。理论上，AngularGrad在收敛性方面具有与Adam相同的遗憾界。然而，在基准数据集上针对最新方法进行的大量实验表明，AngularGrad具有更优越的性能。源代码将在以下网址公开：https://github.com/mhaut/AngularGrad。