Retrieval-augmented language models (RALMs) improve performance by accessing long-tail and up-to-date knowledge from external data stores, but are challenging to build. Existing approaches require either expensive retrieval-specific modifications to LM pre-training or use post-hoc integration of the data store that leads to suboptimal performance. We introduce Retrieval-Augmented Dual Instruction Tuning (RA-DIT), a lightweight fine-tuning methodology that provides a third option by retrofitting any LLM with retrieval capabilities. Our approach operates in two distinct fine-tuning steps: (1) one updates a pre-trained LM to better use retrieved information, while (2) the other updates the retriever to return more relevant results, as preferred by the LM. By fine-tuning over tasks that require both knowledge utilization and contextual awareness, we demonstrate that each stage yields significant performance improvements, and using both leads to additional gains. Our best model, RA-DIT 65B, achieves state-of-the-art performance across a range of knowledge-intensive zero- and few-shot learning benchmarks, significantly outperforming existing in-context RALM approaches by up to +8.9% in 0-shot setting and +1.4% in 5-shot setting on average.

翻译：检索增强型语言模型通过从外部数据存储中获取长尾知识和最新知识来提升性能，但其构建过程具有挑战性。现有方法要么需要对语言模型预训练进行昂贵的检索特定修改，要么采用导致次优性能的数据存储事后集成。我们提出检索增强的双指令微调（RA-DIT），这是一种轻量级微调方法，通过为任意大型语言模型赋予检索能力提供了第三种选择。该方法包含两个独立的微调步骤：（1）更新预训练语言模型以更有效利用检索信息；（2）更新检索器以返回语言模型更偏好的相关性更强结果。通过在需要知识利用与上下文感知的任务上进行微调，我们证明每个阶段均带来显著的性能提升，而联合使用两者可产生额外增益。我们的最佳模型RA-DIT 65B在多项知识密集型零样本和少样本学习基准测试中达到了最先进性能，在零样本设置下平均比现有上下文内检索增强方法提升高达+8.9%，在五样本设置下平均提升+1.4%。