Large Language Models (LLMs)-based text retrieval retrieves documents relevant to search queries based on vector similarities. Documents are pre-encoded offline, while queries arrive in real-time, necessitating an efficient online query encoder. Although LLMs significantly enhance retrieval capabilities, serving deeply parameterized LLMs slows down query inference throughput and increases demands for online deployment resources. In this paper, we propose LightRetriever, a novel LLM-based retriever with extremely lightweight query encoders. Our method retains a full-sized LLM for document encoding, but reduces the workload of query encoding to no more than an embedding lookup. Compared to serving a full LLM on an A800 GPU, our method achieves over 1000x speedup in query encoding and over 10x increase in end-to-end retrieval throughput. Extensive experiments on large-scale retrieval benchmarks show that LightRetriever generalizes well across diverse tasks, maintaining an average of 95% retrieval performance.

翻译：基于大语言模型（LLMs）的文本检索通过向量相似度来检索与搜索查询相关的文档。文档可离线预编码，而查询则实时到达，因此需要一个高效的在线查询编码器。尽管大语言模型显著提升了检索能力，但部署深度参数化的大语言模型会降低查询推理吞吐量，并增加在线部署的资源需求。本文提出LightRetriever，一种基于大语言模型的新型检索器，其查询编码器极为轻量。我们的方法保留完整规模的大语言模型用于文档编码，但将查询编码的工作负载降至不超过一次嵌入查找。与在A800 GPU上部署完整大语言模型相比，我们的方法在查询编码上实现了超过1000倍的加速，端到端检索吞吐量提升了超过10倍。在大规模检索基准上的大量实验表明，LightRetriever在不同任务上具有良好的泛化能力，平均保持了95%的检索性能。