Large language models have achieved remarkable success on general NLP tasks, but they may fall short for domain-specific problems. Recently, various Retrieval-Augmented Large Language Models (RALLMs) are proposed to address this shortcoming. However, existing evaluation tools only provide a few baselines and evaluate them on various domains without mining the depth of domain knowledge. In this paper, we address the challenges of evaluating RALLMs by introducing the R-Eval toolkit, a Python toolkit designed to streamline the evaluation of different RAG workflows in conjunction with LLMs. Our toolkit, which supports popular built-in RAG workflows and allows for the incorporation of customized testing data on the specific domain, is designed to be user-friendly, modular, and extensible. We conduct an evaluation of 21 RALLMs across three task levels and two representative domains, revealing significant variations in the effectiveness of RALLMs across different tasks and domains. Our analysis emphasizes the importance of considering both task and domain requirements when choosing a RAG workflow and LLM combination. We are committed to continuously maintaining our platform at https://github.com/THU-KEG/R-Eval to facilitate both the industry and the researchers.

翻译：大语言模型在通用自然语言处理任务上取得了显著成功，但在特定领域问题上可能表现不足。近期，各种检索增强大语言模型被提出以解决这一缺陷。然而，现有评估工具仅提供少量基线模型，并在多个领域进行评估，未能深入挖掘领域知识的深度。本文通过引入R-Eval工具包应对评估RALLMs的挑战，这是一个旨在简化结合大语言模型的不同检索增强生成工作流评估的Python工具包。我们的工具包支持流行的内置RAG工作流，允许在特定领域纳入定制测试数据，其设计注重用户友好性、模块化和可扩展性。我们跨三个任务层级和两个代表性领域对21个RALLMs进行了评估，揭示了不同任务和领域间RALLMs有效性的显著差异。我们的分析强调了在选择RAG工作流与LLM组合时，必须同时考虑任务需求和领域特性。我们将持续维护平台（https://github.com/THU-KEG/R-Eval），以促进工业界和学术界的研究。