Retrieval-augmented generation (RAG) enhances large language models (LLMs) with external knowledge to answer questions more accurately. However, research on evaluating RAG systems-particularly the retriever component-remains limited, as most existing work focuses on single-context retrieval rather than multi-hop queries, where individual contexts may appear irrelevant in isolation but are essential when combined. In this research, we use the HotPotQA, MuSiQue, and SQuAD datasets to simulate a RAG system and compare three LLM-as-judge evaluation strategies, including our proposed Context-Aware Retriever Evaluation (CARE). Our goal is to better understand how multi-hop reasoning can be most effectively evaluated in RAG systems. Experiments with LLMs from OpenAI, Meta, and Google demonstrate that CARE consistently outperforms existing methods for evaluating multi-hop reasoning in RAG systems. The performance gains are most pronounced in models with larger parameter counts and longer context windows, while single-hop queries show minimal sensitivity to context-aware evaluation. Overall, the results highlight the critical role of context-aware evaluation in improving the reliability and accuracy of retrieval-augmented generation systems, particularly in complex query scenarios. To ensure reproducibility, we provide the complete data of our experiments at https://github.com/lorenzbrehme/CARE.

翻译：检索增强生成（RAG）通过引入外部知识来提升大语言模型（LLMs）的问答准确性。然而，针对RAG系统——特别是检索器组件的评估研究仍较为有限，现有工作多聚焦于单上下文检索而非多跳查询。在多跳场景中，单个上下文看似无关，但组合后对推理至关重要。本研究利用HotPotQA、MuSiQue和SQuAD数据集模拟RAG系统，比较了三种基于大语言模型作为裁判的评估策略（包括我们提出的上下文感知检索器评估（CARE）），旨在深入理解如何在RAG系统中有效评估多跳推理。通过采用OpenAI、Meta和Google的大语言模型进行实验，结果表明CARE在多跳推理评估中始终优于现有方法。性能提升在参数量更大、上下文窗口更长的模型中尤为显著，而单跳查询对上下文感知评估的敏感性较低。总体而言，本研究凸显了上下文感知评估在提升检索增强生成系统可靠性与准确性中的关键作用，尤其在复杂查询场景中。为确保可复现性，我们已在https://github.com/lorenzbrehme/CARE 提供完整实验数据。