In this paper, we provide an overview of first-order and second-order variants of the gradient descent method that are commonly used in machine learning. We propose a general framework in which 6 of these variants can be interpreted as different instances of the same approach. They are the vanilla gradient descent, the classical and generalized Gauss-Newton methods, the natural gradient descent method, the gradient covariance matrix approach, and Newton's method. Besides interpreting these methods within a single framework, we explain their specificities and show under which conditions some of them coincide.

翻译：在本文中,我们概述了在机器学习中常用的梯度下降法的一级和二级变体,我们提出了一个总框架,其中6种变体可被解释为同一方法的不同实例,它们是香草梯度下降、古典和通用的高斯-牛顿法、自然梯度下降法、梯度共变矩阵法和牛顿法。除了在单一框架内解释这些方法外,我们还要解释其中的特性,并表明其中某些条件在什么情况下一致。