Given a vector dataset $\mathcal{X}$, a query vector $\vec{x}_q$, graph-based Approximate Nearest Neighbor Search (ANNS) aims to build a proximity graph (PG) as an index of $\mathcal{X}$ and approximately return vectors with minimum distances to $\vec{x}_q$ by searching over the PG index. It suffers from the large-scale $\mathcal{X}$ because a PG with full vectors is too large to fit into the memory, e.g., a billion-scale $\mathcal{X}$ in 128 dimensions would consume nearly 600 GB memory. To solve this, Product Quantization (PQ) integrated graph-based ANNS is proposed to reduce the memory usage, using smaller compact codes of quantized vectors in memory instead of the large original vectors. Existing PQ methods do not consider the important routing features of PG, resulting in low-quality quantized vectors that affect the ANNS's effectiveness. In this paper, we present an end-to-end Routing-guided learned Product Quantization (RPQ) for graph-based ANNS. It consists of (1) a \textit{differentiable quantizer} used to make the standard discrete PQ differentiable to suit for back-propagation of end-to-end learning, (2) a \textit{sampling-based feature extractor} used to extract neighborhood and routing features of a PG, and (3) a \textit{multi-feature joint training module} with two types of feature-aware losses to continuously optimize the differentiable quantizer. As a result, the inherent features of a PG would be embedded into the learned PQ, generating high-quality quantized vectors. Moreover, we integrate our RPQ with the state-of-the-art DiskANN and existing popular PGs to improve their performance. Comprehensive experiments on real-world large-scale datasets (from 1M to 1B) demonstrate RPQ's superiority, e.g., 1.7$\times$-4.2$\times$ improvement on QPS at the same recall@10 of 95\%.

翻译：给定向量数据集$\mathcal{X}$、查询向量$\vec{x}_q$，基于图的近似最近邻搜索（ANNS）旨在构建一个近邻图（PG）作为$\mathcal{X}$的索引，并通过在PG索引上搜索来近似返回与$\vec{x}_q$距离最小的向量。当$\mathcal{X}$规模较大时，该方法面临挑战，因为包含完整向量的PG过大而无法装入内存。例如，一个128维、十亿规模的$\mathcal{X}$将消耗近600 GB内存。为解决此问题，提出了集成乘积量化（PQ）的基于图的ANNS，通过使用量化向量的紧凑编码代替原始大向量来减少内存占用。现有PQ方法未考虑PG的重要路由特征，导致量化向量质量较低，影响ANNS的有效性。本文提出了一种端到端的基于路由引导的学习乘积量化（RPQ）方法用于基于图的ANNS。该方法包括：（1）一个\textit{可微量化器}，用于使标准离散PQ可微，以适应端到端学习的反向传播；（2）一个\textit{基于采样的特征提取器}，用于提取PG的邻域和路由特征；（3）一个\textit{多特征联合训练模块}，通过两种特征感知损失持续优化可微量化器。因此，PG的固有特征将被嵌入到学习的PQ中，生成高质量的量化向量。此外，我们将RPQ与最先进的DiskANN及现有流行PG集成，以提升其性能。在真实世界大规模数据集（从1M到1B）上的全面实验证明了RPQ的优越性，例如，在召回率@10为95%时，QPS提升了1.7倍至4.2倍。