Retrieval-augmented language models (RALMs) improve performance by accessing long-tail and up-to-date knowledge from external data stores, but are challenging to build. Existing approaches require either expensive retrieval-specific modifications to LM pre-training or use post-hoc integration of the data store that leads to suboptimal performance. We introduce Retrieval-Augmented Dual Instruction Tuning (RA-DIT), a lightweight fine-tuning methodology that provides a third option by retrofitting any LLM with retrieval capabilities. Our approach operates in two distinct fine-tuning steps: (1) one updates a pre-trained LM to better use retrieved information, while (2) the other updates the retriever to return more relevant results, as preferred by the LM. By fine-tuning over tasks that require both knowledge utilization and contextual awareness, we demonstrate that each stage yields significant performance improvements, and using both leads to additional gains. Our best model, RA-DIT 65B, achieves state-of-the-art performance across a range of knowledge-intensive zero- and few-shot learning benchmarks, significantly outperforming existing in-context RALM approaches by up to +8.9% in 0-shot setting and +1.4% in 5-shot setting on average.

翻译：检索增强语言模型（RALMs）通过访问外部数据存储中的长尾知识和最新信息来提升性能，但其构建过程颇具挑战性。现有方法要么需要在语言模型预训练阶段进行昂贵的检索特定修改，要么采用事后集成数据存储的方式导致性能欠佳。我们提出检索增强的双指令微调（RA-DIT）——一种轻量级微调方法，通过为任意大语言模型（LLM）赋加检索能力提供了第三种方案。该方法包含两个独立的微调步骤：（1）更新预训练语言模型以更有效地利用检索信息；（2）更新检索器以返回语言模型更偏好的相关结果。通过在需要知识利用与上下文感知能力的任务上进行微调，我们证明每个步骤都能带来显著的性能提升，而两者协同能产生额外增益。我们的最佳模型RA-DIT 65B在多项知识密集型零样本和少样本学习基准测试中达到领先水平，在零样本设置下平均比现有上下文中RALM方法提升高达+8.9%，在五样本设置下平均提升+1.4%。