In our work, we propose a novel yet simple approach to obtain an adaptive learning rate for gradient-based descent methods on classification tasks. Instead of the traditional approach of selecting adaptive learning rates via the decayed expectation of gradient-based terms, we use the angle between the current gradient and the new gradient: this new gradient is computed from the direction orthogonal to the current gradient, which further helps us in determining a better adaptive learning rate based on angle history, thereby, leading to relatively better accuracy compared to the existing state-of-the-art optimizers. On a wide variety of benchmark datasets with prominent image classification architectures such as ResNet, DenseNet, EfficientNet, and VGG, we find that our method leads to the highest accuracy in most of the datasets. Moreover, we prove that our method is convergent.

翻译：在本文中，我们提出了一种新颖而简单的方法，用于在分类任务中为基于梯度的下降方法获取自适应学习率。与通过梯度项的衰减期望来选择自适应学习率的传统方法不同，我们利用当前梯度与新梯度之间的夹角：这个新梯度是从与当前梯度正交的方向计算得出的，这进一步帮助我们基于角度历史确定更优的自适应学习率，从而相比现有最先进的优化器实现了相对更高的准确率。在采用ResNet、DenseNet、EfficientNet和VGG等主流图像分类架构的广泛基准数据集上，我们发现我们的方法在大多数数据集上达到了最高准确率。此外，我们证明了该方法的收敛性。