Prompt leakage in large language models (LLMs) poses a significant security and privacy threat, particularly in retrieval-augmented generation (RAG) systems. However, leakage in multi-turn LLM interactions along with mitigation strategies has not been studied in a standardized manner. This paper investigates LLM vulnerabilities against prompt leakage across 4 diverse domains and 10 closed- and open-source LLMs. Our unique multi-turn threat model leverages the LLM's sycophancy effect and our analysis dissects task instruction and knowledge leakage in the LLM response. In a multi-turn setting, our threat model elevates the average attack success rate (ASR) to 86.2%, including a 99% leakage with GPT-4 and claude-1.3. We find that some black-box LLMs like Gemini show variable susceptibility to leakage across domains - they are more likely to leak contextual knowledge in the news domain compared to the medical domain. Our experiments measure specific effects of 6 black-box defense strategies, including a query-rewriter in the RAG scenario. Our proposed multi-tier combination of defenses still has an ASR of 5.3% for black-box LLMs, indicating room for enhancement and future direction for LLM security research.

翻译：大型语言模型（LLM）中的提示泄露构成了重大的安全与隐私威胁，尤其在检索增强生成（RAG）系统中。然而，针对多轮LLM交互中的泄露问题及其缓解策略，目前尚未有标准化的研究。本文系统地探究了LLM在4个不同领域和10个闭源与开源模型中的提示泄露脆弱性。我们提出的独特多轮威胁模型利用了LLM的谄媚效应，并深入分析了LLM响应中的任务指令泄露与知识泄露。在多轮设定下，该威胁模型将平均攻击成功率（ASR）提升至86.2%，其中包括针对GPT-4和claude-1.3达到99%的泄露率。我们发现部分黑盒LLM（如Gemini）在不同领域呈现出差异化的泄露敏感性——在新闻领域较医疗领域更易泄露上下文知识。实验量化了6种黑盒防御策略的具体效果，包括RAG场景中的查询重写器。即使采用我们提出的多层组合防御方案，黑盒LLM的ASR仍达5.3%，这表明LLM安全研究仍有改进空间和未来发展方向。