Prompt tuning provides an efficient way for users to customize Large Language Models (LLMs) with their private data in the emerging LLM service scenario. However, the sensitive nature of private data brings the need for privacy preservation in LLM service customization. Based on prompt tuning, we propose Privacy-Preserving Prompt Tuning (RAPT), a framework that provides privacy guarantees for LLM services. \textsc{rapt} adopts a local privacy setting, allowing users to privatize their data locally with local differential privacy. As prompt tuning performs poorly when directly trained on privatized data, we introduce a novel privatized token reconstruction task that is trained jointly with the downstream task, allowing LLMs to learn better task-dependent representations. Despite the simplicity of our framework, experiments show that RAPT achieves competitive performance across tasks while providing privacy guarantees against adversaries.

翻译：提示调优为新兴大语言模型服务场景中用户使用私有数据定制大语言模型提供了一种高效方式。然而，私有数据的敏感性要求在大语言模型服务定制过程中实现隐私保护。基于提示调优，我们提出了一种名为隐私保护提示调优（RAPT）的框架，该框架为大语言模型服务提供隐私保障。RAPT采用本地隐私设置，允许用户通过本地差分隐私在本地对其数据进行私有化处理。针对提示调优在直接使用私有化数据训练时表现不佳的问题，我们引入了一种新颖的私有化Token重建任务，该任务与下游任务联合训练，使大语言模型能够学习到更好的任务相关表示。尽管我们的框架设计简洁，但实验表明，RAPT在提供针对对手的隐私保障的同时，跨任务均取得了具有竞争力的性能。