Retrieval-Augmented Generation (RAG) has become a cornerstone of knowledge-intensive applications, including enterprise chatbots, healthcare assistants, and agentic memory management. However, recent studies show that knowledge-extraction attacks can recover sensitive knowledge-base content through maliciously crafted queries, raising serious concerns about intellectual property theft and privacy leakage. While prior work has explored individual attack and defense techniques, the research landscape remains fragmented, spanning heterogeneous retrieval embeddings, diverse generation models, and evaluations based on non-standardized metrics and inconsistent datasets. To address this gap, we introduce the first systematic benchmark for knowledge-extraction attacks on RAG systems. Our benchmark covers a broad spectrum of attack and defense strategies, representative retrieval embedding models, and both open- and closed-source generators, all evaluated under a unified experimental framework with standardized protocols across multiple datasets. By consolidating the experimental landscape and enabling reproducible, comparable evaluation, this benchmark provides actionable insights and a practical foundation for developing privacy-preserving RAG systems in the face of emerging knowledge extraction threats. Our code is available here.

翻译：检索增强生成（RAG）已成为知识密集型应用的基石，包括企业聊天机器人、医疗健康助手以及智能体记忆管理等。然而，近期研究表明，知识抽取攻击可通过恶意构造的查询恢复敏感知识库内容，引发了关于知识产权盗窃与隐私泄露的严重关切。尽管先前研究已探索了个别的攻击与防御技术，但该领域的研究格局仍较为零散，涉及异构的检索嵌入模型、多样化的生成模型，以及基于非标准化指标与不一致数据集的评估。为弥补这一空白，我们首次提出了针对RAG系统知识抽取攻击的系统性基准测试。该基准涵盖了广泛的攻击与防御策略、具有代表性的检索嵌入模型、开源与闭源生成器，并在统一实验框架下通过跨多个数据集的标准化协议进行评估。通过整合实验环境并实现可复现、可比较的评估，本基准为应对新兴的知识抽取威胁、开发隐私保护的RAG系统提供了可操作的见解与实践基础。我们的代码已在此公开。