A Retrieval-Augmented Language Model (RALM) augments a generative language model by retrieving context-specific knowledge from an external database. This strategy facilitates impressive text generation quality even with smaller models, thus reducing orders of magnitude of computational demands. However, RALMs introduce unique system design challenges due to (a) the diverse workload characteristics between LM inference and retrieval and (b) the various system requirements and bottlenecks for different RALM configurations such as model sizes, database sizes, and retrieval frequencies. We propose Chameleon, a heterogeneous accelerator system that integrates both LM and retrieval accelerators in a disaggregated architecture. The heterogeneity ensures efficient acceleration of both LM inference and retrieval, while the accelerator disaggregation enables the system to independently scale both types of accelerators to fulfill diverse RALM requirements. Our Chameleon prototype implements retrieval accelerators on FPGAs and assigns LM inference to GPUs, with a CPU server orchestrating these accelerators over the network. Compared to CPU-based and CPU-GPU vector search systems, Chameleon achieves up to 23.72x speedup and 26.2x energy efficiency. Evaluated on various RALMs, Chameleon exhibits up to 2.16x reduction in latency and 3.18x speedup in throughput compared to the hybrid CPU-GPU architecture. These promising results pave the way for bringing accelerator heterogeneity and disaggregation into future RALM systems.

翻译：摘要：检索增强语言模型通过从外部数据库中检索特定上下文知识来增强生成式语言模型。该策略即使在使用较小模型时也能促进令人印象深刻的文本生成质量，从而将计算需求降低数个数量级。然而，检索增强语言模型引入了独特的系统设计挑战，原因在于：（a）语言模型推理与检索之间的工作负载特征差异；（b）不同检索增强语言模型配置（如模型规模、数据库规模及检索频率）对系统需求与瓶颈的多样化影响。我们提出Chameleon，这是一种采用解聚架构集成语言模型加速器与检索加速器的异构加速器系统。异构性确保了对语言模型推理与检索两者的高效加速，而加速器解聚使系统能够独立扩展两类加速器以满足多样化的检索增强语言模型需求。我们的Chameleon原型在FPGA上实现检索加速器，将语言模型推理分配给GPU，并由CPU服务器通过网络协调这些加速器。与基于CPU及CPU-GPU的向量搜索系统相比，Chameleon实现了最高23.72倍的加速比和26.2倍的能效提升。在多种检索增强语言模型上的评估表明，与混合CPU-GPU架构相比，Chameleon的延迟降低最高达2.16倍，吞吐量加速比最高达3.18倍。这些令人鼓舞的结果为将加速器异构性与解聚引入未来检索增强语言模型系统铺平了道路。