Retrieval-Augmented Generation (RAG) is a promising approach for mitigating the hallucination of large language models (LLMs). However, existing research lacks rigorous evaluation of the impact of retrieval-augmented generation on different large language models, which make it challenging to identify the potential bottlenecks in the capabilities of RAG for different LLMs. In this paper, we systematically investigate the impact of Retrieval-Augmented Generation on large language models. We analyze the performance of different large language models in 4 fundamental abilities required for RAG, including noise robustness, negative rejection, information integration, and counterfactual robustness. To this end, we establish Retrieval-Augmented Generation Benchmark (RGB), a new corpus for RAG evaluation in both English and Chinese. RGB divides the instances within the benchmark into 4 separate testbeds based on the aforementioned fundamental abilities required to resolve the case. Then we evaluate 6 representative LLMs on RGB to diagnose the challenges of current LLMs when applying RAG. Evaluation reveals that while LLMs exhibit a certain degree of noise robustness, they still struggle significantly in terms of negative rejection, information integration, and dealing with false information. The aforementioned assessment outcomes indicate that there is still a considerable journey ahead to effectively apply RAG to LLMs.

翻译：检索增强生成（RAG）是一种缓解大型语言模型（LLM）幻觉问题的有前景方法。然而，现有研究缺乏对不同大型语言模型在检索增强生成影响下的严谨评估，导致难以识别不同LLM在RAG能力中的潜在瓶颈。本文系统研究了检索增强生成对大型语言模型的影响。我们分析了不同大型语言模型在RAG所需的四项基本能力上的表现，包括噪声鲁棒性、负向拒绝、信息整合和反事实鲁棒性。为此，我们建立了检索增强生成基准（RGB），这是一个用于英语和中文RAG评估的新语料库。RGB根据解决案例所需的上述基本能力，将基准中的实例划分为四个独立测试平台。随后，我们在RGB上评估了6个代表性LLM，以诊断当前LLM在应用RAG时面临的挑战。评估结果显示，尽管LLM表现出一定程度的噪声鲁棒性，但在负向拒绝、信息整合和处理虚假信息方面仍存在显著困难。上述评估结果表明，将RAG有效应用于LLM仍有相当长的路要走。