Retrieval-augmented language models (RALMs) improve performance by accessing long-tail and up-to-date knowledge from external data stores, but are challenging to build. Existing approaches require either expensive retrieval-specific modifications to LM pre-training or use post-hoc integration of the data store that leads to suboptimal performance. We introduce Retrieval-Augmented Dual Instruction Tuning (RA-DIT), a lightweight fine-tuning methodology that provides a third option by retrofitting any LLM with retrieval capabilities. Our approach operates in two distinct fine-tuning steps: (1) one updates a pre-trained LM to better use retrieved information, while (2) the other updates the retriever to return more relevant results, as preferred by the LM. By fine-tuning over tasks that require both knowledge utilization and contextual awareness, we demonstrate that each stage yields significant performance improvements, and using both leads to additional gains. Our best model, RA-DIT 65B, achieves state-of-the-art performance across a range of knowledge-intensive zero- and few-shot learning benchmarks, significantly outperforming existing in-context RALM approaches by up to +8.9% in 0-shot setting and +1.4% in 5-shot setting on average.

翻译：检索增强型语言模型通过访问外部数据存储中的长尾和最新知识来提升性能，但其构建过程颇具挑战。现有方法要么需要对语言模型预训练进行昂贵的检索特定修改，要么采用后验式数据存储集成导致性能欠佳。我们提出检索增强双重指令调优（RA-DIT），这是一种轻量级微调方法，通过为任意大语言模型赋予检索能力提供了第三种方案。该方法包含两个不同的微调步骤：（1）更新预训练语言模型以更有效地利用检索信息，（2）更新检索器以返回语言模型更偏好的相关结果。通过在同时需要知识运用和语境感知能力的任务上进行微调，我们证明每个阶段都能带来显著的性能提升，而联合使用两者则可获得额外增益。我们的最佳模型RA-DIT 65B在多个知识密集型零样本和少样本学习基准上取得了最先进的性能，在零样本设置下平均比现有基于上下文学习的检索增强语言模型方法提升高达+8.9%，在五样本设置下平均提升+1.4%。