Evaluating retrieval-augmented generation (RAG) presents challenges, particularly for retrieval models within these systems. Traditional end-to-end evaluation methods are computationally expensive. Furthermore, evaluation of the retrieval model's performance based on query-document relevance labels shows a small correlation with the RAG system's downstream performance. We propose a novel evaluation approach, eRAG, where each document in the retrieval list is individually utilized by the large language model within the RAG system. The output generated for each document is then evaluated based on the downstream task ground truth labels. In this manner, the downstream performance for each document serves as its relevance label. We employ various downstream task metrics to obtain document-level annotations and aggregate them using set-based or ranking metrics. Extensive experiments on a wide range of datasets demonstrate that eRAG achieves a higher correlation with downstream RAG performance compared to baseline methods, with improvements in Kendall's $\tau$ correlation ranging from 0.168 to 0.494. Additionally, eRAG offers significant computational advantages, improving runtime and consuming up to 50 times less GPU memory than end-to-end evaluation.

翻译：检索增强生成（RAG）的评估面临挑战，特别是针对其内部检索模型。传统的端到端评估方法计算成本高昂。此外，基于查询-文档相关性标签对检索模型性能的评估，与RAG系统下游性能之间的相关性较低。我们提出了一种新颖的评估方法eRAG，其中RAG系统内的大语言模型逐一利用检索列表中的每个文档。基于每个文档生成的输出随后根据下游任务真实标签进行评估。通过这种方式，每个文档的下游性能即作为其相关性标签。我们采用多种下游任务指标获取文档级标注，并使用基于集合或排名的指标进行聚合。在广泛数据集上的大量实验表明，与基线方法相比，eRAG与下游RAG性能的相关性更高，Kendall's $\tau$ 相关系数的提升范围在0.168至0.494之间。此外，eRAG具有显著的计算优势，运行时间更短，且GPU内存消耗相较端到端评估最多可降低50倍。