Retrieval-Augmented Generation (RAG) is a state-of-the-art technique that enhances Large Language Models (LLMs) by retrieving relevant knowledge from an external, non-parametric database. This approach aims to mitigate common LLM issues such as hallucinations and outdated knowledge. Although existing research has demonstrated security and privacy vulnerabilities within RAG systems, making them susceptible to attacks like jailbreaks and prompt injections, the security of the RAG system's external databases remains largely underexplored. In this paper, we employ Membership Inference Attacks (MIA) to determine whether a sample is part of the knowledge database of a RAG system, using only black-box API access. Our core hypothesis posits that if a sample is a member, it will exhibit significant similarity to the text generated by the RAG system. To test this, we compute the cosine similarity and the model's perplexity to establish a membership score, thereby building robust features. We then introduce two novel attack strategies: a Threshold-based Attack and a Machine Learning-based Attack, designed to accurately identify membership. Experimental validation of our methods has achieved a ROC AUC of 82%.

翻译：检索增强生成（RAG）是一种前沿技术，它通过从外部非参数化数据库中检索相关知识来增强大型语言模型（LLM）。该方法旨在缓解LLM的常见问题，如幻觉和知识过时。尽管现有研究已揭示了RAG系统中存在的安全与隐私漏洞，使其易受越狱和提示注入等攻击，但RAG系统外部数据库的安全性在很大程度上仍未得到充分探索。本文采用成员推断攻击（MIA），仅通过黑盒API访问，来判断一个样本是否属于RAG系统的知识数据库。我们的核心假设是：如果一个样本是数据库成员，它将与RAG系统生成的文本表现出显著相似性。为验证此假设，我们计算余弦相似度和模型困惑度以建立成员分数，从而构建鲁棒特征。随后，我们提出了两种新颖的攻击策略：基于阈值的攻击和基于机器学习的攻击，旨在精确识别成员身份。实验验证表明，我们的方法取得了82%的ROC AUC。