Graph-based ANNS algorithms have gained increasing research interest and market adoption due to their efficiency and accuracy in retrieval. Existing approaches primarily rely on CPUs for graph index construction and retrieval, but this often requires significant time, especially for large-scale and high-dimensional datasets. Some studies have explored GPU-based solutions. However, GPUs are costly and their limited memory makes handling large datasets challenging. In this paper, we propose a novel end-to-end system ScaleGANN that enables users to efficiently construct graph indexes for large-scale, high-dimensional datasets by leveraging low-cost spot GPU resources in a distributed cloud system. ScaleGANN utilized the idea of divide-and-merge, with an optimized vector partitioning algorithm to further improve the indexing time and space efficiency while guaranteeing good index quality. Its novel resource allocation strategy realized multi-GPU indexing parallelism and overall cost-effectiveness for both build and query. Besides, we designed a task scheduler and cost model for better spot instance management and evaluation. We tested our system on large real-world datasets. Experiment results show that our approach can significantly accelerate the index build time to up to 9x times at even 6x lower price compared with the state-of-the-art extendable ANNS benchmark DiskANN.

翻译：基于图的近似最近邻搜索算法因其在检索中的高效性和准确性而受到越来越多的研究关注和市场采纳。现有方法主要依赖CPU进行图索引构建和检索，但这通常需要大量时间，尤其是在处理大规模和高维数据集时。部分研究探索了基于GPU的解决方案，然而GPU成本高昂且内存有限，使得处理大规模数据集面临挑战。本文提出了一种新颖的端到端系统ScaleGANN，该系统通过利用分布式云系统中的低成本抢占式GPU资源，使用户能够高效构建大规模高维数据集的图索引。ScaleGANN采用分治合并的思想，并结合优化的向量划分算法，在保证索引质量的同时进一步提升索引时间和空间效率。其创新的资源分配策略实现了多GPU索引并行性，并在构建和查询两方面确保了整体成本效益。此外，我们设计了任务调度器和成本模型，以更好地管理抢占式实例并进行评估。我们在大规模真实数据集上测试了该系统。实验结果表明，与当前最先进的可扩展ANNS基准DiskANN相比，我们的方法能够将索引构建时间加速高达9倍，同时成本降低至多6倍。