Large language models are powerful text processors and reasoners, but are still subject to limitations including outdated knowledge and hallucinations, which necessitates connecting them to the world. Retrieval-augmented large language models have raised extensive attention for grounding model generation on external knowledge. However, retrievers struggle to capture relevance, especially for queries with complex information needs. Recent work has proposed to improve relevance modeling by having large language models actively involved in retrieval, i.e., to improve retrieval with generation. In this paper, we show that strong performance can be achieved by a method we call Iter-RetGen, which synergizes retrieval and generation in an iterative manner. A model output shows what might be needed to finish a task, and thus provides an informative context for retrieving more relevant knowledge which in turn helps generate a better output in the next iteration. Compared with recent work which interleaves retrieval with generation when producing an output, Iter-RetGen processes all retrieved knowledge as a whole and largely preserves the flexibility in generation without structural constraints. We evaluate Iter-RetGen on multi-hop question answering, fact verification, and commonsense reasoning, and show that it can flexibly leverage parametric knowledge and non-parametric knowledge, and is superior to or competitive with state-of-the-art retrieval-augmented baselines while causing fewer overheads of retrieval and generation. We can further improve performance via generation-augmented retrieval adaptation.

翻译：大语言模型作为强大的文本处理器和推理器，仍受限于知识过时和幻觉等问题，需将其与外部世界连接。检索增强型大语言模型通过将模型生成锚定于外部知识而备受关注。然而，检索器难以捕捉相关性，尤其是在处理具有复杂信息需求的查询时。近期研究提出通过让大语言模型主动参与检索（即利用生成改进检索）来提升相关性建模。本文证明，我们提出的Iter-RetGen方法可通过迭代方式协同检索与生成实现强性能。模型输出揭示了完成任务所需内容，为检索更相关知识提供信息上下文，进而帮助生成下一轮更优输出。与近期在生成过程中交错检索与生成的工作不同，Iter-RetGen将所有检索知识视为整体进行处理，最大程度保留无结构约束的生成灵活性。我们在多跳问答、事实验证和常识推理任务上评估了Iter-RetGen，表明其能灵活利用参数化与非参数化知识，在减少检索与生成开销的同时优于或匹敌最先进的检索增强基线模型。通过生成增强的检索适应性，可进一步改善性能。