Retrieval-augmented language models (RALMs) improve performance by accessing long-tail and up-to-date knowledge from external data stores, but are challenging to build. Existing approaches require either expensive retrieval-specific modifications to LM pre-training or use post-hoc integration of the data store that leads to suboptimal performance. We introduce Retrieval-Augmented Dual Instruction Tuning (RA-DIT), a lightweight fine-tuning methodology that provides a third option by retrofitting any LLM with retrieval capabilities. Our approach operates in two distinct fine-tuning steps: (1) one updates a pre-trained LM to better use retrieved information, while (2) the other updates the retriever to return more relevant results, as preferred by the LM. By fine-tuning over tasks that require both knowledge utilization and contextual awareness, we demonstrate that each stage yields significant performance improvements, and using both leads to additional gains. Our best model, RA-DIT 65B, achieves state-of-the-art performance across a range of knowledge-intensive zero- and few-shot learning benchmarks, significantly outperforming existing in-context RALM approaches by up to +8.9% in 0-shot setting and +1.4% in 5-shot setting on average.

翻译：检索增强语言模型通过访问外部数据存储中的长尾知识和最新信息来提升性能，但其构建过程颇具挑战性。现有方法要么需要在对语言模型进行预训练时引入昂贵的检索专用修改，要么采用事后整合数据存储的方式导致次优性能。我们提出检索增强的双重指令微调（RA-DIT），这是一种轻量级微调方法，通过为任意大型语言模型赋予检索能力提供了第三种选择。该方法包含两个独立的微调步骤：（1）更新预训练语言模型以更好地利用检索到的信息；（2）更新检索器以返回语言模型更偏好的相关结果。通过在需要知识利用和上下文感知能力的任务上进行微调，我们证明每个阶段都能带来显著的性能提升，而结合使用两者可产生协同增益。我们最优模型RA-DIT 65B在一系列知识密集型零样本和少样本学习基准测试中实现了最先进的性能，在零样本设置下平均显著优于现有上下文内检索增强语言模型方法达+8.9%，在五样本设置下平均提升+1.4%。