Retrieval-Augmented Generation (RAG) is a technique that enhances the capabilities of large language models (LLMs) by incorporating external knowledge sources. This method addresses common LLM limitations, including outdated information and the tendency to produce inaccurate "hallucinated" content. However, the evaluation of RAG systems is challenging, as existing benchmarks are limited in scope and diversity. Most of the current benchmarks predominantly assess question-answering applications, overlooking the broader spectrum of situations where RAG could prove advantageous. Moreover, they only evaluate the performance of the LLM component of the RAG pipeline in the experiments, and neglect the influence of the retrieval component and the external knowledge database. To address these issues, this paper constructs a large-scale and more comprehensive benchmark, and evaluates all the components of RAG systems in various RAG application scenarios. Specifically, we have categorized the range of RAG applications into four distinct types-Create, Read, Update, and Delete (CRUD), each representing a unique use case. "Create" refers to scenarios requiring the generation of original, varied content. "Read" involves responding to intricate questions in knowledge-intensive situations. "Update" focuses on revising and rectifying inaccuracies or inconsistencies in pre-existing texts. "Delete" pertains to the task of summarizing extensive texts into more concise forms. For each of these CRUD categories, we have developed comprehensive datasets to evaluate the performance of RAG systems. We also analyze the effects of various components of the RAG system, such as the retriever, the context length, the knowledge base construction, and the LLM. Finally, we provide useful insights for optimizing the RAG technology for different scenarios.

翻译：检索增强生成（RAG）是一种通过引入外部知识源来增强大型语言模型（LLM）能力的技术。该方法可解决LLM的常见局限性，包括信息过时及产生不准确的"幻觉"内容。然而，RAG系统的评估颇具挑战性，现有基准在范围和多样性方面均存在局限。当前大多数基准主要评估问答应用场景，忽略了RAG可能发挥优势的更广泛情境。此外，它们仅在实验中评估RAG流水线中LLM组件的性能，而忽视了检索组件与外部知识数据库的影响。针对这些问题，本文构建了一个大规模、更全面的基准，并在多种RAG应用场景中对系统所有组件进行了评估。具体而言，我们将RAG应用范围划分为四种不同类型——创建（Create）、读取（Read）、更新（Update）与删除（Delete），每种类型代表一种独特用例。"创建"指需要生成原创多样化内容的场景；"读取"涉及在知识密集型情境中回答复杂问题；"更新"聚焦于修正和纠正已有文本中的错误或不一致之处；"删除"则涉及将长文本归纳为更简洁形式的任务。针对每个CRUD类别，我们开发了全面的数据集以评估RAG系统性能。同时分析了检索器、上下文长度、知识库构建及LLM等RAG系统各组件的效果。最后，我们为针对不同场景优化RAG技术提供了有益见解。