Large pretrained language models (LLMs) have shown surprising In-Context Learning (ICL) ability. An important application in deploying large language models is to augment LLMs with a private database for some specific task. The main problem with this promising commercial use is that LLMs have been shown to memorize their training data and their prompt data are vulnerable to membership inference attacks (MIA) and prompt leaking attacks. In order to deal with this problem, we treat LLMs as untrusted in privacy and propose a locally differentially private framework of in-context learning(LDP-ICL) in the settings where labels are sensitive. Considering the mechanisms of in-context learning in Transformers by gradient descent, we provide an analysis of the trade-off between privacy and utility in such LDP-ICL for classification. Moreover, we apply LDP-ICL to the discrete distribution estimation problem. In the end, we perform several experiments to demonstrate our analysis results.

翻译：大型预训练语言模型展现了令人惊讶的上下文学习能力。在部署大型语言模型时，一个重要应用是为特定任务增强其私有数据库。这一具有商业前景的应用面临的主要问题是：语言模型会记忆其训练数据，其提示数据易受成员推理攻击和提示泄露攻击。为解决该问题，我们将语言模型视为不可信隐私实体，并在标签敏感场景下提出局部差分隐私的上下文学习框架。考虑Transformer中通过梯度下降实现的上下文学习机制，我们分析了此类分类任务在隐私与效用间的权衡。此外，我们将所提框架应用于离散分布估计问题。最后通过多项实验验证了理论分析结果。