This paper introduces Stochastic RAG--a novel approach for end-to-end optimization of retrieval-augmented generation (RAG) models that relaxes the simplifying assumptions of marginalization and document independence, made in most prior work. Stochastic RAG casts the retrieval process in RAG as a stochastic sampling without replacement process. Through this formulation, we employ straight-through Gumbel-top-k that provides a differentiable approximation for sampling without replacement and enables effective end-to-end optimization for RAG. We conduct extensive experiments on seven diverse datasets on a wide range of tasks, from open-domain question answering to fact verification to slot-filling for relation extraction and to dialogue systems. By applying this optimization method to a recent and effective RAG model, we advance state-of-the-art results on six out of seven datasets.

翻译：本文提出Stochastic RAG——一种用于检索增强生成模型端到端优化的新方法，该方法放宽了先前大多数工作中所采用的边际化和文档独立性简化假设。Stochastic RAG将RAG中的检索过程建模为无放回随机采样过程。通过这一建模方式，我们采用straight-through Gumbel-top-k技术，该技术为无放回采样提供了可微近似，从而实现了RAG模型的有效端到端优化。我们在涵盖开放域问答、事实核查、关系抽取的槽填充任务以及对话系统等广泛任务的七个多样化数据集上进行了大量实验。通过将这种优化方法应用于近期高效RAG模型，我们在七个数据集中的六个上取得了最先进的成果。