Large Language Models (LLMs) are increasingly integrated with external tools. While these integrations can significantly improve the functionality of LLMs, they also create a new attack surface where confidential data may be disclosed between different components. Specifically, malicious tools can exploit vulnerabilities in the LLM itself to manipulate the model and compromise the data of other services, raising the question of how private data can be protected in the context of LLM integrations. In this work, we provide a systematic way of evaluating confidentiality in LLM-integrated systems. For this, we formalize a "secret key" game that can capture the ability of a model to conceal private information. This enables us to compare the vulnerability of a model against confidentiality attacks and also the effectiveness of different defense strategies. In this framework, we evaluate eight previously published attacks and four defenses. We find that current defenses lack generalization across attack strategies. Building on this analysis, we propose a method for robustness fine-tuning, inspired by adversarial training. This approach is effective in lowering the success rate of attackers and in improving the system's resilience against unknown attacks.

翻译：大型语言模型（LLM）正越来越多地与外部工具集成。虽然这些集成可以显著提升LLM的功能性，但同时也引入了新的攻击面，使得机密数据可能在不同组件之间泄露。具体而言，恶意工具可利用LLM自身的漏洞来操纵模型，并危及其他服务的数据安全，这引发了如何在LLM集成环境中保护私有数据的问题。在本工作中，我们提供了一种系统化评估LLM集成系统中机密性的方法。为此，我们形式化了一种“密钥”游戏，能够捕捉模型隐藏私有信息的能力。这使得我们能够比较模型面对机密性攻击的脆弱性，以及不同防御策略的有效性。在此框架下，我们评估了八种先前发表的攻击方法和四种防御策略。我们发现，当前的防御方法缺乏应对不同攻击策略的泛化能力。基于这一分析，我们提出了一种受对抗训练启发的鲁棒性微调方法。该方法能有效降低攻击者的成功率，并提升系统对未知攻击的抵御能力。