We prove closed-form equations for the exact high-dimensional asymptotics of a family of first order gradient-based methods, learning an estimator (e.g. M-estimator, shallow neural network, ...) from observations on Gaussian data with empirical risk minimization. This includes widely used algorithms such as stochastic gradient descent (SGD) or Nesterov acceleration. The obtained equations match those resulting from the discretization of dynamical mean-field theory (DMFT) equations from statistical physics when applied to gradient flow. Our proof method allows us to give an explicit description of how memory kernels build up in the effective dynamics, and to include non-separable update functions, allowing datasets with non-identity covariance matrices. Finally, we provide numerical implementations of the equations for SGD with generic extensive batch-size and with constant learning rates.

翻译：我们证明了一类基于梯度的初等方法在高维极限下的精确闭式方程，这类方法通过经验风险最小化从高斯数据观测中学习估计器（如M估计器、浅层神经网络等），其中包括随机梯度下降（SGD）和内斯特罗夫加速等广泛使用的算法。所得方程与统计物理学中梯度流动力学平均场理论（DMFT）方程离散化结果一致。我们的证明方法能够显式描述记忆核在有效动力学中的构建过程，并包含非可分离更新函数，从而适用于协方差矩阵非恒等的数据集。最后，我们针对具有通用大批量大小和恒定学习率的SGD算法给出了方程数值实现。