A convergence analysis is developed for the regularized Newton method for training neural networks (NNs) in the overparameterized limit. As the number of hidden units tends to infinity, the NN training dynamics converge in probability to the solution of a deterministic limit equation involving a ``Newton neural tangent kernel'' (NNTK). Explicit rates characterizing this convergence are provided and, in the infinite-width limit, we prove that the NN converges exponentially fast to the target data (i.e., a global minimizer with zero loss). We show that this convergence is uniform across the frequency spectrum, addressing the spectral bias inherent in gradient descent. The eigenvalues of the NTK for gradient descent accumulate at zero, leading to slow convergence for target data with high-frequency components. In contrast, the NNTK has uniformly lower bounded eigenvalues if the regularization parameter is selected appropriately, allowing Newton's method to converge more quickly for data with high-frequency components. Mathematical challenges that need to be addressed in our analysis include the implicit parameter update of the Newton method with a potentially indefinite Hessian matrix and the fact that the dimension of this linear system of equations tends to infinity as the NN width grows. This complicates deriving the training dynamics in the overparameterized limit as well as proving the convergence of the finite-width dynamics thereto. The analysis identifies a scaling formula for selecting the regularization parameter, which we show can vanish at a suitable rate as the number of hidden units becomes larger. We prove that, for sufficiently large numbers of hidden units, the regularized Hessian remains positive definite during training and the Newton updates for individual NN parameters converge to zero, showing that the model behaves as a linearization around the initialization.

翻译：针对过参数化极限下训练神经网络的正则化牛顿法，本文发展了收敛性分析。当隐单元数量趋于无穷时，神经网络训练动态以概率收敛到涉及“牛顿神经正切核”（NNTK）的确定性极限方程的解。我们给出了刻画该收敛的显式速率，并在无穷宽极限下证明了神经网络指数级收敛到目标数据（即零损失的全局极小点）。我们证明该收敛在频谱上是均匀的，解决了梯度下降固有的谱偏差问题。梯度下降的NTK特征值在零处累积，导致含高频成分的目标数据收敛缓慢。相比之下，若适当选择正则化参数，NNTK具有一致下界的特征值，使得牛顿法对含高频成分的数据收敛更快。分析中需处理的数学挑战包括：可能具有不定Hessian矩阵的牛顿法的隐式参数更新，以及该线性方程组的维数随神经网络宽度增长而趋于无穷。这为导出过参数化极限下的训练动态以及证明有限宽度动态收敛于此极限增加了难度。分析给出了选择正则化参数的缩放公式，我们证明该参数可随隐单元数量增加而以适当速率趋于零。我们证明：对于足够大的隐单元数量，正则化Hessian矩阵在训练过程中保持正定，且单个神经网络参数的牛顿更新收敛到零，表明模型行为类似于初始化处的线性化。