Retrieval-augmented generation (RAG) extends large language models (LLMs) with external data sources to enhance factual correctness and domain coverage. Modern RAG pipelines rely on large datastores, creating a significant system challenge: achieving high throughput and low latency is difficult, especially when GPU memory is limited. To address these challenges, we propose TeleRAG, an efficient inference system that reduces latency and improves throughput with minimal GPU memory requirements. The core innovation of TeleRAG is lookahead retrieval, a prefetching mechanism that predicts required data and transfers them from CPU to GPU in parallel with LLM generation. In addition, TeleRAG adopts a prefetching scheduler and a cache-aware scheduler to support efficient multi-GPU inference with minimal overhead. Evaluations show TeleRAG achieves up to a 1.53x average end-to-end latency reduction (single-query) and 1.83x higher average throughput (batched), as well as good scalability in throughput. This confirms the practical utility of TeleRAG for faster and more memory-efficient deployments of RAG applications.

翻译：检索增强生成（RAG）通过整合外部数据源扩展大型语言模型（LLMs），以提升事实准确性与领域覆盖能力。现代RAG流水线依赖大规模数据存储，这带来了显著的系统挑战：在GPU内存受限的情况下，实现高吞吐量与低延迟尤为困难。为应对这些挑战，我们提出TeleRAG——一种高效推理系统，能以最小GPU内存需求降低延迟并提升吞吐量。TeleRAG的核心创新是前瞻检索机制，该预取机制通过预测所需数据，在LLM生成过程中并行将数据从CPU传输至GPU。此外，TeleRAG采用预取调度器与缓存感知调度器，以最小开销支持高效的多GPU推理。评估结果表明，TeleRAG在单查询场景下平均端到端延迟降低最高达1.53倍，批处理场景下平均吞吐量提升最高达1.83倍，并展现出良好的吞吐量可扩展性。这证实了TeleRAG在实现更快速、更高内存效率的RAG应用部署方面具有实际效用。