A Retrieval-Augmented Language Model (RALM) augments a generative language model by retrieving context-specific knowledge from an external database. This strategy facilitates impressive text generation quality even with smaller models, thus reducing orders of magnitude of computational demands. However, RALMs introduce unique system design challenges due to (a) the diverse workload characteristics between LM inference and retrieval and (b) the various system requirements and bottlenecks for different RALM configurations such as model sizes, database sizes, and retrieval frequencies. We propose Chameleon, a heterogeneous accelerator system that integrates both LM and retrieval accelerators in a disaggregated architecture. The heterogeneity ensures efficient acceleration of both LM inference and retrieval, while the accelerator disaggregation enables the system to independently scale both types of accelerators to fulfill diverse RALM requirements. Our Chameleon prototype implements retrieval accelerators on FPGAs and assigns LM inference to GPUs, with a CPU server orchestrating these accelerators over the network. Compared to CPU-based and CPU-GPU vector search systems, Chameleon achieves up to 23.72x speedup and 26.2x energy efficiency. Evaluated on various RALMs, Chameleon exhibits up to 2.16x reduction in latency and 3.18x speedup in throughput compared to the hybrid CPU-GPU architecture. These promising results pave the way for bringing accelerator heterogeneity and disaggregation into future RALM systems.

翻译：检索增强语言模型（RALM）通过从外部数据库中检索特定上下文知识来增强生成式语言模型。即使使用较小的模型，该策略也能促进令人印象深刻的文本生成质量，从而将计算需求降低数个数量级。然而，RALM引入了独特的系统设计挑战，原因在于：（a）语言模型推理与检索之间的多样化工作负载特性；（b）不同RALM配置（如模型大小、数据库大小和检索频率）带来的不同系统需求和瓶颈。我们提出Chameleon，一种采用解耦架构的异构加速器系统，集成了语言模型与检索加速器。异构性确保了语言模型推理与检索的高效加速，而加速器解耦则使系统能够独立扩展两种类型的加速器，以满足多样化的RALM需求。我们的Chameleon原型在FPGA上实现检索加速器，将语言模型推理分配给GPU，并由CPU服务器通过网络协调这些加速器。与基于CPU和CPU-GPU的向量搜索系统相比，Chameleon实现高达23.72倍的加速比和26.2倍的能效提升。在各种RALM上评估，与混合CPU-GPU架构相比，Chameleon在延迟上降低高达2.16倍，在吞吐量上实现高达3.18倍的加速。这些有前景的结果为将加速器异构性与解耦引入未来RALM系统铺平了道路。