Dense passage retrieval (DPR) is the first step in the retrieval augmented generation (RAG) paradigm for improving the performance of large language models (LLM). DPR fine-tunes pre-trained networks to enhance the alignment of the embeddings between queries and relevant textual data. A deeper understanding of DPR fine-tuning will be required to fundamentally unlock the full potential of this approach. In this work, we explore DPR-trained models mechanistically by using a combination of probing, layer activation analysis, and model editing. Our experiments show that DPR training decentralizes how knowledge is stored in the network, creating multiple access pathways to the same information. We also uncover a limitation in this training style: the internal knowledge of the pre-trained model bounds what the retrieval model can retrieve. These findings suggest a few possible directions for dense retrieval: (1) expose the DPR training process to more knowledge so more can be decentralized, (2) inject facts as decentralized representations, (3) model and incorporate knowledge uncertainty in the retrieval process, and (4) directly map internal model knowledge to a knowledge base.

翻译：密集段落检索（DPR）是检索增强生成（RAG）范式中提升大语言模型（LLM）性能的第一步。DPR通过对预训练网络进行微调，以增强查询与相关文本数据之间嵌入表示的对齐程度。要充分发挥该方法的潜力，需要深入理解DPR微调的本质。本研究通过结合探针分析、层激活分析和模型编辑，对DPR训练后的模型进行机制性探究。实验表明，DPR训练使得知识在网络中的存储方式呈去中心化分布，为相同信息创建了多条访问路径。同时我们发现该训练方式存在局限性：预训练模型的内部知识边界限制了检索模型的实际检索能力。这些发现为密集检索提出了几个可能方向：（1）向DPR训练过程注入更多知识以增强去中心化程度；（2）将事实以去中心化表征形式注入模型；（3）在检索过程中建模并整合知识不确定性；（4）将模型内部知识直接映射至知识库。