As retrieval-augmented generation (RAG) becomes more widespread, the role of retrieval is shifting from retrieving information for human browsing to retrieving context for AI reasoning. This shift creates more complex search environments, where relevance is difficult to pre-define. Existing retrievers rely on supervised fine-tuning (SFT) with human labels or synthetic data, resulting in static relevance that struggles to adapt to diverse RAG environments. To address this challenge, we propose R3, a Retrieval framework optimized for RAG through Reinforcement learning (RL). Specifically, we adopt an RL training paradigm that enables the retriever to explore and self-improve within given RAG environments, automating the learning process with minimal manual experimentation or tuning effort. Extensive experiments across diverse tasks demonstrate that R3 improves RAG performance by 5.2% over the original retriever and surpasses state-of-the-art retrievers by 4.9%, while achieving comparable results to LLM-augmented retrieval and RAG systems built on post-trained or instruction-tuned LLMs. It is both efficient and practical, requiring only 4 GPUs and completing training within a single day.

翻译：随着检索增强生成（RAG）的日益普及，检索的角色正从为人类浏览检索信息转变为为AI推理检索上下文。这一转变催生了更为复杂的搜索环境，其中相关性难以预先定义。现有的检索器依赖于带有人工标注或合成数据的监督微调（SFT），导致其相关性是静态的，难以适应多样化的RAG环境。为应对这一挑战，我们提出了R3，一个通过强化学习（RL）为RAG优化的检索框架。具体而言，我们采用了一种RL训练范式，使检索器能够在给定的RAG环境中进行探索和自我改进，从而以最少的人工实验或调优工作实现学习过程的自动化。在多样化任务上的大量实验表明，R3将RAG性能相较于原始检索器提升了5.2%，并超越了最先进的检索器4.9%，同时达到了与基于后训练或指令微调的大型语言模型（LLM）构建的LLM增强检索及RAG系统相当的结果。该方法高效且实用，仅需4个GPU并在一天内即可完成训练。