The Retrieval Augmented Generation (RAG) framework utilizes a combination of parametric knowledge and external knowledge to demonstrate state-of-the-art performance on open-domain question answering tasks. However, the RAG framework suffers from performance degradation when the query is accompanied by irrelevant contexts. In this work, we propose the RE-RAG framework, which introduces a relevance estimator (RE) that not only provides relative relevance between contexts as previous rerankers did, but also provides confidence, which can be used to classify whether given context is useful for answering the given question. We propose a weakly supervised method for training the RE simply utilizing question-answer data without any labels for correct contexts. We show that RE trained with a small generator (sLM) can not only improve the sLM fine-tuned together with RE but also improve previously unreferenced large language models (LLMs). Furthermore, we investigate new decoding strategies that utilize the proposed confidence measured by RE such as choosing to let the user know that it is "unanswerable" to answer the question given the retrieved contexts or choosing to rely on LLM's parametric knowledge rather than unrelated contexts.

翻译：检索增强生成（RAG）框架结合参数化知识与外部知识，在开放域问答任务上展现了最先进的性能。然而，当查询伴随不相关上下文时，RAG框架会出现性能下降。本文提出RE-RAG框架，引入一个相关性估计器（RE），该估计器不仅如先前重排序器那样提供上下文之间的相对相关性，还能提供置信度，用于判断给定上下文是否对回答给定问题有用。我们提出一种弱监督方法来训练RE，仅利用问答数据而无需任何正确上下文的标注。实验表明，使用小型生成器（sLM）训练的RE不仅能提升与RE共同微调的sLM的性能，还能改进先前未引用的大型语言模型（LLM）。此外，我们探索了利用RE所提供置信度的新解码策略，例如选择告知用户基于检索到的上下文“无法回答”问题，或选择依赖LLM的参数化知识而非不相关的上下文。