We investigate the connections between sparse approximation methods for making kernel methods and Gaussian processes (GPs) scalable to large-scale data, focusing on the Nystr\"om method and the Sparse Variational Gaussian Processes (SVGP). While sparse approximation methods for GPs and kernel methods share some algebraic similarities, the literature lacks a deep understanding of how and why they are related. This may pose an obstacle to the communications between the GP and kernel communities, making it difficult to transfer results from one side to the other. Our motivation is to remove this obstacle, by clarifying the connections between the sparse approximations for GPs and kernel methods. In this work, we study the two popular approaches, the Nystr\"om and SVGP approximations, in the context of a regression problem, and establish various connections and equivalences between them. In particular, we provide an RKHS interpretation of the SVGP approximation, and show that the Evidence Lower Bound of the SVGP contains the objective function of the Nystr\"om approximation, revealing the origin of the algebraic equivalence between the two approaches. We also study recently established convergence results for the SVGP and how they are related to the approximation quality of the Nystr\"om method.

翻译：我们研究了稀疏近似方法在使核方法和高斯过程（GPs）适用于大规模数据时的联系，重点探讨Nyström方法与稀疏变分高斯过程（SVGP）。尽管高斯过程和核方法的稀疏近似方法在代数上具有一定相似性，但文献中缺乏对它们之间关联方式及原因的深入理解。这可能阻碍GP与核方法领域之间的交流，使得跨领域的研究成果难以相互转化。我们的动机是通过阐明GP与核方法中稀疏近似之间的联系来消除这一障碍。本研究以回归问题为背景，深入分析了Nyström和SVGP两种主流近似方法，并建立了它们之间的多种联系与等价关系。特别地，我们给出了SVGP近似的再生核希尔伯特空间（RKHS）解释，并证明SVGP的证据下界包含了Nyström近似的目标函数，揭示了两者代数等价的根源。此外，我们还研究了近期建立的SVGP收敛性结果及其与Nyström方法近似质量之间的关联。