Optimization problems that include regularization functions in their objectives are regularly solved in many applications. When one seeks second-order methods for such problems, it may be desirable to exploit specific properties of some of these regularization functions when accounting for curvature information in the solution steps to speed up convergence. In this paper, we propose the SCORE (self-concordant regularization) framework for unconstrained minimization problems which incorporates second-order information in the Newton-decrement framework for convex optimization. We propose the generalized Gauss-Newton with Self-Concordant Regularization (GGN-SCORE) algorithm that updates the minimization variables each time it receives a new input batch. The proposed algorithm exploits the structure of the second-order information in the Hessian matrix, thereby reducing computational overhead. GGN-SCORE demonstrates how to speed up convergence while also improving model generalization for problems that involve regularized minimization under the proposed SCORE framework. Numerical experiments show the efficiency of our method and its fast convergence, which compare favorably against baseline first-order and quasi-Newton methods. Additional experiments involving non-convex (overparameterized) neural network training problems show that the proposed method is promising for non-convex optimization.

翻译：许多应用场景中频繁求解包含正则化函数的目标优化问题。当针对这类问题寻求二阶方法时，利用某些正则化函数在求解步骤中处理曲率信息的特定性质，有望加速收敛。本文针对无约束极小化问题提出SCORE（自和谐正则化）框架，该框架在凸优化的牛顿减量框架中融入了二阶信息。我们提出广义高斯-牛顿自和谐正则化（GGN-SCORE）算法，该算法在每次接收新输入批次时更新极小化变量。所提算法利用了海森矩阵中二阶信息的结构，从而降低了计算开销。GGN-SCORE展示了如何在加速收敛的同时改善涉及SCORE框架下正则化极小化问题的模型泛化能力。数值实验表明该方法具有高效性与快速收敛性，其性能优于基线一阶方法和拟牛顿法。在非凸（过参数化）神经网络训练问题上的补充实验表明，所提方法在非凸优化领域具有应用前景。