As the state-of-the-art methods for high-dimensional data retrieval, Approximate Nearest Neighbor Search (ANNS) approaches with graph-based indexes have attracted increasing attention and play a crucial role in many real-world applications, e.g., retrieval-augmented generation (RAG) and recommendation systems. Unlike the extensive works focused on designing efficient graph-based ANNS methods, this paper delves into merging multiple existing graph-based indexes into a single one, which is also crucial in many real-world scenarios (e.g., cluster consolidation in distributed systems and read-write contention in real-time vector databases). We propose a Fast Graph-based Indexes Merging (FGIM) framework with three core techniques: (1) Proximity Graphs (PGs) to $k$ Nearest Neighbor Graph ($k$-NNG) transformation used to extract potential candidate neighbors from input graph-based indexes through cross-querying, (2) $k$-NNG refinement designed to identify overlooked high-quality neighbors and maintain graph connectivity, and (3) $k$-NNG to PG transformation aimed at improving graph navigability and enhancing search performance. Then, we integrate our FGIM framework with the state-of-the-art ANNS method, HNSW, and other existing mainstream graph-based methods to demonstrate its generality and merging efficiency. Extensive experiments on six real-world datasets show that our FGIM framework is applicable to various mainstream graph-based ANNS methods, achieves up to 3.5$\times$ speedup over HNSW's incremental construction and an average of 7.9$\times$ speedup for methods without incremental support, while maintaining comparable or superior search performance.
翻译:作为高维数据检索的先进方法,基于图索引的近似最近邻搜索(ANNS)方法日益受到关注,并在众多实际应用中发挥关键作用,例如检索增强生成(RAG)和推荐系统。与大量专注于设计高效图基ANNS方法的研究不同,本文深入探讨将多个现有图索引合并为单一索引的问题,这在实际场景(如分布式系统中的集群整合和实时向量数据库中的读写竞争)中也至关重要。我们提出了一种快速的基于图的索引合并(FGIM)框架,包含三项核心技术:(1)近邻图(PG)到k近邻图(k-NNG)的转换,用于通过交叉查询从输入图索引中提取潜在候选邻居;(2)k-NNG精化,旨在识别被忽视的高质量邻居并维持图的连通性;以及(3)k-NNG到PG的转换,旨在提升图的可导航性和搜索性能。接着,我们将FGIM框架与当前最先进的ANNS方法HNSW以及其他现有主流图方法集成,以展示其通用性和合并效率。在六个真实数据集上的广泛实验表明,我们的FGIM框架适用于多种主流的基于图的ANNS方法,与HNSW的增量构建相比可加速高达3.5倍,对于不支持增量构建的方法平均加速7.9倍,同时保持相当或更优的搜索性能。