The growing use of large language model (LLM)-based conversational agents to manage sensitive user data raises significant privacy concerns. While these agents excel at understanding and acting on context, this capability can be exploited by malicious actors. We introduce a novel threat model where adversarial third-party apps manipulate the context of interaction to trick LLM-based agents into revealing private information not relevant to the task at hand. Grounded in the framework of contextual integrity, we introduce AirGapAgent, a privacy-conscious agent designed to prevent unintended data leakage by restricting the agent's access to only the data necessary for a specific task. Extensive experiments using Gemini, GPT, and Mistral models as agents validate our approach's effectiveness in mitigating this form of context hijacking while maintaining core agent functionality. For example, we show that a single-query context hijacking attack on a Gemini Ultra agent reduces its ability to protect user data from 94% to 45%, while an AirGapAgent achieves 97% protection, rendering the same attack ineffective.

翻译：基于大语言模型的对话代理在管理敏感用户数据方面的应用日益增长，引发了显著的隐私担忧。尽管这类代理擅长理解上下文并据此采取行动，但这种能力可能被恶意行为者利用。我们提出了一种新型威胁模型：恶意第三方应用通过操纵交互上下文，诱使基于大语言模型的代理泄露与当前任务无关的私人信息。基于情境完整性理论框架，我们引入了AirGapAgent——一种隐私敏感型代理，通过将代理访问权限限制为特定任务必需的数据，来防止意外数据泄露。使用Gemini、GPT和Mistral模型作为代理的广泛实验验证了该方法在缓解此类上下文劫持攻击的同时，保持核心代理功能的有效性。例如，针对Gemini Ultra代理的单次查询上下文劫持攻击，使其保护用户数据的能力从94%降至45%，而AirGapAgent实现了97%的保护率，使同一攻击失效。