Despite the impressive numerical performance of the quasi-Newton and Anderson/nonlinear acceleration methods, their global convergence rates have remained elusive for over 50 years. This study addresses this long-standing issue by introducing a framework that derives novel, adaptive quasi-Newton and nonlinear/Anderson acceleration schemes. Under mild assumptions, the proposed iterative methods exhibit explicit, non-asymptotic convergence rates that blend those of the gradient descent and Cubic Regularized Newton's methods. The proposed approach also includes an accelerated version for convex functions. Notably, these rates are achieved adaptively without prior knowledge of the function's parameters. The framework presented in this study is generic, and its special cases includes algorithms such as Newton's method with random subspaces, finite-differences, or lazy Hessian. Numerical experiments demonstrated the efficiency of the proposed framework, even compared to the l-BFGS algorithm with Wolfe line-search.

翻译：尽管拟牛顿法和安德森/非线性加速方法在数值性能上表现卓越，但其全局收敛速率问题在过去五十余年中始终未获解答。本研究通过引入一个新型框架，推导出自适应拟牛顿法和非线性/安德森加速方案，从而解决了这一长期难题。在温和假设条件下，所提出的迭代方法表现出显式的非渐近收敛速率，该速率融合了梯度下降法和三次正则牛顿法的收敛特性。针对凸函数，该框架还包含加速版本。值得注意的是，这些收敛速率可自适应实现，无需预先获知函数参数。本研究提出的框架具有通用性，其特例涵盖随机子空间牛顿法、有限差分牛顿法及惰性海森矩阵牛顿法等算法。数值实验表明，即便与采用Wolfe线搜索的l-BFGS算法相比，所提框架仍展现出高效性。