Sketch-and-project is a framework which unifies many known iterative methods for solving linear systems and their variants, as well as further extensions to non-linear optimization problems. It includes popular methods such as randomized Kaczmarz, coordinate descent, variants of the Newton method in convex optimization, and others. In this paper, we develop a theoretical framework for obtaining sharp guarantees on the convergence rate of sketch-and-project methods. Our approach is the first to: (1) show that the convergence rate improves at least linearly with the sketch size, and even faster when the data matrix exhibits certain spectral decays; and (2) allow for sparse sketching matrices, which are more efficient than dense sketches and more robust than sub-sampling methods. In particular, our results explain an observed phenomenon that a radical sparsification of the sketching matrix does not affect the per iteration convergence rate of sketch-and-project. To obtain our results, we develop new non-asymptotic spectral bounds for the expected sketched projection matrix, which are of independent interest; and we establish a connection between the convergence rates of iterative sketch-and-project solvers and the approximation error of randomized singular value decomposition, which is a widely used one-shot sketching algorithm for low-rank approximation. Our experiments support the theory and demonstrate that even extremely sparse sketches exhibit the convergence properties predicted by our framework.

翻译：投影-素描是一个统一多种已知迭代方法（用于线性系统求解及其变体，以及非线性优化问题的扩展）的框架。它涵盖随机卡茨马尔兹法、坐标下降法、凸优化中牛顿法的变体等流行方法。本文建立了一个理论框架，用于获得投影-素描方法收敛速率的严格保证。我们的方法是首个：(1) 证明收敛速率至少随素描尺寸线性提升，且当数据矩阵具有特定谱衰减时提升更快；(2) 允许稀疏素描矩阵，其效率高于密集素描且比子采样方法更鲁棒。特别地，我们的结果解释了观测现象：素描矩阵的极端稀疏化不影响投影-素描每次迭代的收敛速率。为获得结果，我们开发了期望素描投影矩阵的新非渐近谱界（这具有独立意义），并建立了迭代投影-素描求解器收敛速率与随机奇异值分解近似误差之间的连系——后者是一种广泛用于低秩近似的单步素描算法。我们的实验支持该理论，并表明即使极端稀疏的素描也能展现出预测的收敛性质。