Randomized Kaczmarz methods form a family of linear system solvers which converge by repeatedly projecting their iterates onto randomly sampled equations. While effective in some contexts, such as highly over-determined least squares, Kaczmarz methods are traditionally deemed secondary to Krylov subspace methods, since this latter family of solvers can exploit outliers in the input's singular value distribution to attain fast convergence on ill-conditioned systems. In this paper, we introduce Kaczmarz++, an accelerated randomized block Kaczmarz algorithm that exploits outlying singular values in the input to attain a fast Krylov-style convergence. Moreover, we show that Kaczmarz++ captures large outlying singular values provably faster than popular Krylov methods, for both over- and under-determined systems. We also develop an optimized variant for positive semidefinite systems, called CD++, demonstrating empirically that it is competitive in arithmetic operations with both CG and GMRES on a collection of benchmark problems. To attain these results, we introduce several novel algorithmic improvements to the Kaczmarz framework, including adaptive momentum acceleration, Tikhonov-regularized projections, and a memoization scheme for reusing information from previously sampled equation~blocks.

翻译：随机化Kaczmarz方法是一类通过将迭代解重复投影到随机选取的方程上来实现收敛的线性系统求解器。尽管在某些场景（如高度超定最小二乘问题）中表现优异，传统上Kaczmarz方法被视为Krylov子空间方法的次级选择，因为后者能够利用输入矩阵奇异值分布中的离群值，在病态系统上实现快速收敛。本文提出Kaczmarz++算法——一种加速的随机分块Kaczmarz方法，该算法通过利用输入矩阵中的离群奇异值实现了类Krylov的快速收敛。我们进一步证明，对于超定与欠定系统，Kaczmarz++在理论上能比主流Krylov方法更快速地捕获显著的离群奇异值。针对半正定系统，我们还开发了优化变体CD++，实验表明该算法在基准问题集上的算术运算复杂度与共轭梯度法（CG）及广义最小残差法（GMRES）具有竞争力。为实现这些结果，我们对Kaczmarz框架引入了多项创新性算法改进，包括自适应动量加速、Tikhonov正则化投影，以及用于复用历史采样方程块信息的记忆化方案。