Approximate Nearest-Neighbor Search (ANNS) is a key technique in retrieval-augmented generation (RAG), enabling rapid identification of the most relevant high-dimensional embeddings from massive vector databases. Modern ANNS engines accelerate this process using prebuilt indexes and store compressed vector-quantized representations in fast memory. However, they still rely on a costly second-pass refinement stage that reads full-precision vectors from slower storage like SSDs. For modern text and multimodal embeddings, these reads now dominate the latency of the entire query. We propose FaTRQ, a far-memory-aware refinement system using tiered memory that eliminates the need to fetch full vectors from storage. It introduces a progressive distance estimator that refines coarse scores using compact residuals streamed from far memory. Refinement stops early once a candidate is provably outside the top-k. To support this, we propose tiered residual quantization, which encodes residuals as ternary values stored efficiently in far memory. A custom accelerator is deployed in a CXL Type-2 device to perform low-latency refinement locally. Together, FaTRQ improves the storage efficiency by 2.4$\times$ and improves the throughput by up to 9$ \times$ than SOTA GPU ANNS system.

翻译：近似最近邻搜索是检索增强生成中的关键技术，能够从海量向量数据库中快速识别最相关的高维嵌入。现代ANNS引擎通过预构建索引加速此过程，并将压缩的向量量化表示存储在快速内存中。然而，它们仍依赖昂贵的二次精炼阶段，需要从SSD等慢速存储中读取全精度向量。对于现代文本与多模态嵌入，这类读取操作已成为整个查询延迟的主要瓶颈。本文提出FaTRQ——一种采用分层内存的远内存感知精炼系统，无需从存储中获取完整向量。该系统引入渐进式距离估计器，利用从远内存流式传输的紧凑残差对粗略分数进行精炼。当候选向量被证明位于前k名之外时，精炼过程将提前终止。为此，我们提出分层残差量化技术，将残差编码为三元数值并高效存储在远内存中。通过部署在CXL Type-2设备中的定制加速器，可在本地执行低延迟精炼。综合而言，FaTRQ相比最先进的GPU ANNS系统，存储效率提升2.4倍，吞吐量最高提升9倍。