Though second-order optimization methods are highly effective, popular approaches in machine learning such as SGD and Adam use only first-order information due to the difficulty of computing curvature in high dimensions. We present FOSI, a novel meta-algorithm that improves the performance of any first-order optimizer by efficiently incorporating second-order information during the optimization process. In each iteration, FOSI implicitly splits the function into two quadratic functions defined on orthogonal subspaces, then uses a second-order method to minimize the first, and the base optimizer to minimize the other. Our analysis of FOSI's preconditioner and effective Hessian proves that FOSI improves the condition number for a large family of optimizers. Our empirical evaluation demonstrates that FOSI improves the convergence rate and optimization time of GD, Heavy-Ball, and Adam when applied to several deep neural networks training tasks such as audio classification, transfer learning, and object classification and when applied to convex functions.

翻译：尽管二阶优化方法效果显著，但机器学习中常用的SGD和Adam等主流方法仅利用一阶信息，原因在于高维空间中曲率计算的困难性。本文提出FOSI——一种新型元算法，通过高效融合二阶信息以提升任意一阶优化器的性能。在每次迭代中，FOSI将目标函数隐式分解为定义于正交子空间上的两个二次函数，随后分别采用二阶方法最小化前者，并利用基优化器最小化后者。通过分析FOSI的预条件子与有效海森矩阵，我们证明该方法可在广泛的优化器家族中改善条件数。实验评估表明，在音频分类、迁移学习、目标分类等深度神经网络训练任务以及凸函数优化中，FOSI能显著提升梯度下降法(GD)、重型球法(Heavy-Ball)与Adam的收敛速度与优化时间。