Non-asymptotic convergence analysis of quasi-Newton methods has gained attention with a landmark result establishing an explicit local superlinear rate of O$((1/\sqrt{t})^t)$. The methods that obtain this rate, however, exhibit a well-known drawback: they require the storage of the previous Hessian approximation matrix or all past curvature information to form the current Hessian inverse approximation. Limited-memory variants of quasi-Newton methods such as the celebrated L-BFGS alleviate this issue by leveraging a limited window of past curvature information to construct the Hessian inverse approximation. As a result, their per iteration complexity and storage requirement is O$(\tau d)$ where $\tau\le d$ is the size of the window and $d$ is the problem dimension reducing the O$(d^2)$ computational cost and memory requirement of standard quasi-Newton methods. However, to the best of our knowledge, there is no result showing a non-asymptotic superlinear convergence rate for any limited-memory quasi-Newton method. In this work, we close this gap by presenting a Limited-memory Greedy BFGS (LG-BFGS) method that can achieve an explicit non-asymptotic superlinear rate. We incorporate displacement aggregation, i.e., decorrelating projection, in post-processing gradient variations, together with a basis vector selection scheme on variable variations, which greedily maximizes a progress measure of the Hessian estimate to the true Hessian. Their combination allows past curvature information to remain in a sparse subspace while yielding a valid representation of the full history. Interestingly, our established non-asymptotic superlinear convergence rate demonstrates an explicit trade-off between the convergence speed and memory requirement, which to our knowledge, is the first of its kind. Numerical results corroborate our theoretical findings and demonstrate the effectiveness of our method.

翻译：拟牛顿方法的非渐近收敛性分析因一项里程碑式结果而备受关注，该结果建立了显式的局部超线性收敛速率 O$((1/\sqrt{t})^t)$。然而，实现该速率的方法存在众所周知的问题：它们需要存储之前的海森矩阵近似或所有历史曲率信息，以构建当前海森逆矩阵近似。拟牛顿方法的有限记忆变体（如著名的L-BFGS）通过利用有限窗口的历史曲率信息来构造海森逆矩阵近似，从而缓解了这一问题。因此，其每步迭代复杂度和存储需求为 O$(\tau d)$（其中 $\tau\le d$ 为窗口大小，$d$ 为问题维度），显著降低了标准拟牛顿方法的 O$(d^2)$ 计算开销和内存需求。但据我们所知，尚无研究证明任何有限记忆拟牛顿方法具有非渐近超线性收敛速率。本文通过提出一种有限记忆贪心BFGS（LG-BFGS）方法填补了这一空白，该方法能实现显式的非渐近超线性收敛速率。我们创新性地融合了位移聚合（即解相关投影）对梯度变化的后处理方法，以及基于变量变化的选择基向量机制——该机制通过最大化海森矩阵估计与真实海森矩阵之间的进步度量来实现贪心选取。二者的结合使历史曲率信息在稀疏子空间中得以保留，同时生成完整历史过程的有效表征。引人注目的是，我们建立的非渐近超线性收敛速率首次揭示了收敛速度与内存需求之间的显式权衡关系（据我们所知为领域首创）。数值实验结果验证了理论发现，并展示了所提方法的有效性。