The rise of large language models (LLMs) has significantly transformed both the construction and application of information retrieval (IR) systems. However, current interactions between IR systems and LLMs remain limited, with LLMs merely serving as part of components within IR systems, and IR systems being constructed independently of LLMs. This separated architecture restricts knowledge sharing and deep collaboration between them. In this paper, we introduce Self-Retrieval, a novel end-to-end LLM-driven information retrieval architecture. Self-Retrieval unifies all essential IR functions within a single LLM, leveraging the inherent capabilities of LLMs throughout the IR process. Specifically, Self-Retrieval internalizes the retrieval corpus through self-supervised learning, transforms the retrieval process into sequential passage generation, and performs relevance assessment for reranking. Experimental results demonstrate that Self-Retrieval not only outperforms existing retrieval approaches by a significant margin, but also substantially enhances the performance of LLM-driven downstream applications like retrieval-augmented generation.

翻译：大型语言模型（LLM）的兴起显著改变了信息检索（IR）系统的构建与应用方式。然而，当前IR系统与LLM之间的交互仍存在局限：LLM仅作为IR系统的组成部分发挥作用，而IR系统的构建也独立于LLM。这种分离式架构限制了两者间的知识共享与深度协同。本文提出自检索（Self-Retrieval）——一种新颖的端到端LLM驱动信息检索架构。该架构将IR核心功能统一整合于单一LLM中，充分利用LLM在IR全流程中的内在能力。具体而言，自检索通过自监督学习将检索语料库内化于模型，将检索过程转化为序列化段落生成任务，并执行相关性评估以实现重排序。实验结果表明，自检索不仅显著超越现有检索方法，还能大幅提升检索增强生成等LLM驱动下游应用的性能。