Training and inference in Gaussian processes (GPs) require solving linear systems with $n\times n$ kernel matrices. To address the prohibitive $\mathcal{O}(n^3)$ time complexity, recent work has employed fast iterative methods, like conjugate gradients (CG). However, as datasets increase in magnitude, the kernel matrices become increasingly ill-conditioned and still require $\mathcal{O}(n^2)$ space without partitioning. Thus, while CG increases the size of datasets GPs can be trained on, modern datasets reach scales beyond its applicability. In this work, we propose an iterative method which only accesses subblocks of the kernel matrix, effectively enabling mini-batching. Our algorithm, based on alternating projection, has $\mathcal{O}(n)$ per-iteration time and space complexity, solving many of the practical challenges of scaling GPs to very large datasets. Theoretically, we prove the method enjoys linear convergence. Empirically, we demonstrate its fast convergence in practice and robustness to ill-conditioning. On large-scale benchmark datasets with up to four million data points, our approach accelerates GP training and inference by speed-up factors up to $27\times$ and $72 \times$, respectively, compared to CG.

翻译：在高斯过程（GP）的训练与推理中，需要求解涉及 $n\times n$ 核矩阵的线性系统。为应对 $\mathcal{O}(n^3)$ 时间复杂度的挑战，近期研究采用了共轭梯度法（CG）等快速迭代方法。然而，随着数据规模增大，核矩阵的条件数恶化且未经分块时仍需要 $\mathcal{O}(n^2)$ 存储空间。因此，尽管CG扩展了GP可训练的数据集规模，但现代数据集已达到其适用范围之外。本文提出一种仅访问核矩阵子块的迭代方法，有效实现了小批量处理。基于交替投影的算法具有每轮迭代 $\mathcal{O}(n)$ 的时间与空间复杂度，解决了GP扩展到超大数据集的诸多实际难题。理论上，我们证明了该算法具有线性收敛性。实验表明，该方法在实践中的快速收敛能力及其对病态问题的鲁棒性。在包含多达四百万数据点的大规模基准数据集上，与CG相比，我们的方法可将GP训练与推理速度分别提升高达 $27\times$ 和 $72\times$。