Retrieval-augmented large language models (LLMs) leverage relevant content retrieved by information retrieval systems to generate correct responses, aiming to alleviate the hallucination problem. However, existing retriever-responder methods typically append relevant documents to the prompt of LLMs to perform text generation tasks without considering the interaction of fine-grained structural semantics between the retrieved documents and the LLMs. This issue is particularly important for accurate response generation as LLMs tend to "lose in the middle" when dealing with input prompts augmented with lengthy documents. In this work, we propose a new pipeline named "Reinforced Retriever-Reorder-Responder" (R$^4$) to learn document orderings for retrieval-augmented LLMs, thereby further enhancing their generation abilities while the large numbers of parameters of LLMs remain frozen. The reordering learning process is divided into two steps according to the quality of the generated responses: document order adjustment and document representation enhancement. Specifically, document order adjustment aims to organize retrieved document orderings into beginning, middle, and end positions based on graph attention learning, which maximizes the reinforced reward of response quality. Document representation enhancement further refines the representations of retrieved documents for responses of poor quality via document-level gradient adversarial learning. Extensive experiments demonstrate that our proposed pipeline achieves better factual question-answering performance on knowledge-intensive tasks compared to strong baselines across various public datasets. The source codes and trained models will be released upon paper acceptance.

翻译：检索增强型大语言模型通过利用信息检索系统获取的相关内容来生成准确响应，旨在缓解幻觉问题。然而，现有的检索器-响应器方法通常将相关文档直接附加至大语言模型的提示词中执行文本生成任务，未能充分考虑检索文档与大语言模型之间细粒度结构语义的交互。该问题对于精确响应生成尤为重要，因为当输入提示词附带冗长文档时，大语言模型往往会出现"中间信息丢失"现象。本研究提出名为"强化检索器-重排序-响应器"的新流程框架，通过为检索增强型大语言模型学习文档排序策略，在保持大语言模型海量参数冻结的前提下进一步提升其生成能力。重排序学习过程根据生成响应的质量分为两个步骤：文档顺序调整与文档表征增强。具体而言，文档顺序调整基于图注意力学习机制，将检索文档按起始、中间和末端位置进行组织，以最大化响应质量的强化奖励信号。文档表征增强则通过文档级梯度对抗学习，针对低质量响应进一步优化检索文档的表征。大量实验表明，相较于各公共数据集上的强基线模型，我们提出的流程框架在知识密集型任务上实现了更优的事实性问答性能。源代码与训练模型将在论文录用后公开发布。