Dense retrieval needs to learn discriminative text embeddings to represent the semantic relationship between query and document. It may benefit from the using of large language models (LLMs), given LLMs' strong capability on semantic understanding. However, the LLMs are pre-trained by text generation tasks, whose working pattern is completely different from representing texts as embeddings. As a result, it is imperative to study how to adapt LLMs properly so that they can be effectively initialized as the backbone encoder for dense retrieval. In this paper, we propose a novel approach, called LLaRA (LLM adapted for dense RetrievAl), which works as a post-hoc adaptation of LLM for the dense retrieval application. LLaRA consists of two pretext tasks: EBAE (Embedding-Based Auto-Encoding) and EBAR (Embedding-Based Auto-Regression), where the text embeddings from LLM are used to reconstruct the tokens for the input sentence and predict the tokens for the next sentence, respectively. LLaRA turns out to be simple, lightweight, and highly effective. It is applied to adapt LLaMA-2-7B (base) on the Wikipedia corpus, where it substantially improves the model's fine-tuned performances on a variety of dense retrieval benchmarks, like MSMARCO and BEIR. Our model and code will be made publicly available at BGE repository.

翻译：稠密检索需要学习具有区分性的文本嵌入，以表示查询与文档之间的语义关系。鉴于大型语言模型（LLM）在语义理解方面的强大能力，稠密检索可能受益于LLM的运用。然而，LLM是通过文本生成任务进行预训练的，其工作模式与将文本表示为嵌入的方式完全不同。因此，如何合理适配LLM，使其能够有效初始化为稠密检索的骨干编码器，成为一个亟待研究的问题。本文提出了一种名为LLaRA（面向稠密检索适配的LLM）的新方法，该方法作为LLM在稠密检索应用中的事后适配方案。LLaRA包含两个前置任务：EBAE（基于嵌入的自编码）和EBAR（基于嵌入的自回归），分别利用LLM生成的文本嵌入来重构输入句子的令牌及预测下一句子的令牌。LLaRA被证明简单、轻量且高效。我们将其应用于基于维基百科语料库适配LLaMA-2-7B（基础版），该方法显著提升了模型在MSMARCO和BEIR等多个稠密检索基准上的微调性能。我们的模型和代码将公开发布于BGE仓库。