The generalized Gauss-Newton (GGN) optimization method incorporates curvature estimates into its solution steps, and provides a good approximation to the Newton method for large-scale optimization problems. GGN has been found particularly interesting for practical training of deep neural networks, not only for its impressive convergence speed, but also for its close relation with neural tangent kernel regression, which is central to recent studies that aim to understand the optimization and generalization properties of neural networks. This work studies a GGN method for optimizing a two-layer neural network with explicit regularization. In particular, we consider a class of generalized self-concordant (GSC) functions that provide smooth approximations to commonly-used penalty terms in the objective function of the optimization problem. This approach provides an adaptive learning rate selection technique that requires little to no tuning for optimal performance. We study the convergence of the two-layer neural network, considered to be overparameterized, in the optimization loop of the resulting GGN method for a given scaling of the network parameters. Our numerical experiments highlight specific aspects of GSC regularization that help to improve generalization of the optimized neural network. The code to reproduce the experimental results is available at https://github.com/adeyemiadeoye/ggn-score-nn.

翻译：广义高斯-牛顿（GGN）优化方法在求解步骤中融入曲率估计，为大规模优化问题提供了牛顿法的良好近似。GGN在深度神经网络的实践训练中备受关注，不仅因其惊人的收敛速度，还因其与神经正切核回归的紧密联系——后者正是旨在理解神经网络优化与泛化特性的近期研究核心。本文研究了一种含显式正则化的双层神经网络GGN优化方法。具体而言，我们考虑一类广义自和谐（GSC）函数，该类函数可为优化问题目标函数中常用的惩罚项提供光滑近似。该方法提出的自适应学习率选择技术几乎无需调参即可达到最优性能。在给定网络参数缩放条件下，我们研究了过参数化双层神经网络在该GGN方法优化循环中的收敛性。数值实验突出了GSC正则化中能提升优化神经网络泛化能力的具体特征。重现实验结果的代码可于https://github.com/adeyemiadeoye/ggn-score-nn获取。