Large language models (LLMs) and AI agents are increasingly integrated into enterprise systems to access internal databases and generate context-aware responses. While such integration improves productivity and decision support, the model outputs may inadvertently reveal sensitive information. Although many prior efforts focus on protecting the privacy of user prompts, relatively few studies consider privacy risks from the enterprise data perspective. Hence, this paper develops a probabilistic framework for analyzing privacy leakage in AI agents based on differential privacy. We model response generation as a stochastic mechanism that maps prompts and datasets to distributions over token sequences. Within this framework, we introduce token-level and message-level differential privacy and derive privacy bounds that relate privacy leakage to generation parameters such as temperature and message length. We further formulate a privacy-utility design problem that characterizes optimal temperature selection.

翻译：大语言模型（LLM）和AI代理正日益融入企业系统，以访问内部数据库并生成上下文感知的响应。尽管此类集成提升了生产力和决策支持能力，但模型输出可能无意中泄露敏感信息。虽然诸多先前研究聚焦于保护用户提示的隐私，但鲜有研究从企业数据视角考虑隐私风险。为此，本文基于差分隐私开发了一个分析AI代理中隐私泄露的概率框架。我们将响应生成建模为一种随机机制，该机制将提示和数据集映射到token序列的分布上。在该框架内，我们引入了token级和消息级差分隐私，并推导出将隐私泄露与生成参数（如温度和消息长度）相关联的隐私边界。我们进一步构建了一个隐私-效用设计问题，用以刻画最优温度选择。