Estimating the Hessian matrix, especially for neural network training, is a challenging problem due to high dimensionality and cost. In this work, we compare the classical Sherman-Morrison update used in the popular BFGS method (Broy-den-Fletcher-Goldfarb-Shanno), which maintains a positive definite Hessian approximation under a convexity assumption, with a novel approach called Online Gradient Regression (OGR). OGR performs regression of gradients against positions using an exponential moving average to estimate second derivatives online, without requiring Hessian inversion. Unlike BFGS, OGR allows estimation of a general (not necessarily positive definite) Hessian and can thus handle non-convex structures. We evaluate both methods across standard test functions and demonstrate that OGR achieves faster convergence and improved loss, particularly in non-convex settings.

翻译：在神经网络训练等高维且计算代价高昂的场景中，海森矩阵的估计是一个具有挑战性的问题。本研究比较了经典BFGS方法（Broy-den-Fletcher-Goldfarb-Shanno）中采用的Sherman-Morrison更新策略——该方法在凸性假设下保持正定的海森矩阵近似——与一种称为在线梯度回归（Online Gradient Regression, OGR）的新方法。OGR通过指数移动平均对梯度与位置进行回归分析，在线估计二阶导数，无需进行海森矩阵求逆。与BFGS不同，OGR能够估计一般形式（未必正定）的海森矩阵，从而可处理非凸结构。我们在标准测试函数上评估了两种方法，结果表明OGR在非凸场景中尤其表现出更快的收敛速度和更优的损失函数值。