Prompt leakage in large language models (LLMs) poses a significant security and privacy threat, particularly in retrieval-augmented generation (RAG) systems. However, leakage in multi-turn LLM interactions along with mitigation strategies has not been studied in a standardized manner. This paper investigates LLM vulnerabilities against prompt leakage across 4 diverse domains and 10 closed- and open-source LLMs. Our unique multi-turn threat model leverages the LLM's sycophancy effect and our analysis dissects task instruction and knowledge leakage in the LLM response. In a multi-turn setting, our threat model elevates the average attack success rate (ASR) to 86.2%, including a 99% leakage with GPT-4 and claude-1.3. We find that some black-box LLMs like Gemini show variable susceptibility to leakage across domains - they are more likely to leak contextual knowledge in the news domain compared to the medical domain. Our experiments measure specific effects of 6 black-box defense strategies, including a query-rewriter in the RAG scenario. Our proposed multi-tier combination of defenses still has an ASR of 5.3% for black-box LLMs, indicating room for enhancement and future direction for LLM security research.

翻译：大型语言模型（LLM）中的提示泄露构成了重大的安全与隐私威胁，尤其在检索增强生成（RAG）系统中。然而，多轮LLM交互中的泄露现象及其缓解策略尚未以标准化方式进行研究。本文针对4个不同领域的10个闭源与开源LLM，系统考察了其面对提示泄露的脆弱性。我们独特的“多轮威胁模型”利用了LLM的迎合效应，并深入分析了LLM响应中的任务指令泄露与知识泄露。在多轮设置下，该威胁模型将平均攻击成功率（ASR）提升至86.2%，其中GPT-4和Claude-1.3的泄露率高达99%。研究发现，部分黑盒LLM（如Gemini）在不同领域表现出差异化的泄露敏感性——新闻领域的上下文知识泄露概率显著高于医疗领域。实验量化评估了6种黑盒防御策略（包括RAG场景下的查询改写器）的具体效果。我们提出的多层级组合防御方案下，黑盒LLM的ASR仍达5.3%，揭示了LLM安全研究的提升空间与未来方向。