We present an approach for generating differentially private synthetic text using large language models (LLMs), via private prediction. In the private prediction framework, we only require the output synthetic data to satisfy differential privacy guarantees. This is in contrast to approaches that train a generative model on potentially sensitive user-supplied source data and seek to ensure the model itself is safe to release. We prompt a pretrained LLM with source data, but ensure that next-token predictions are made with differential privacy guarantees. Previous work in this paradigm reported generating a small number of examples (<10) at reasonable privacy levels, an amount of data that is useful only for downstream in-context learning or prompting. In contrast, we make changes that allow us to generate thousands of high-quality synthetic data points, greatly expanding the set of potential applications. Our improvements come from an improved privacy analysis and a better private selection mechanism, which makes use of the equivalence between the softmax layer for sampling tokens in LLMs and the exponential mechanism. Furthermore, we introduce a novel use of public predictions via the sparse vector technique, in which we do not pay privacy costs for tokens that are predictable without sensitive data; we find this to be particularly effective for structured data.

翻译：本文提出一种利用大语言模型（LLM）通过隐私预测机制生成差分隐私合成文本的方法。在隐私预测框架中，我们仅要求输出的合成数据满足差分隐私保证，这与在潜在敏感的用户提供源数据上训练生成模型、并试图确保模型本身可安全发布的方法形成对比。我们使用源数据提示预训练的LLM，但通过差分隐私保证进行下一词元预测。该范式下的先前研究仅能在合理隐私级别下生成少量样本（<10个），这类数据量仅适用于下游上下文学习或提示任务。相比之下，我们通过改进实现了数千个高质量合成数据点的生成，极大拓展了潜在应用场景。我们的改进源于更优的隐私分析和更高效的隐私选择机制——该机制利用了LLM采样词元的softmax层与指数机制之间的等价性。此外，我们创新性地通过稀疏向量技术引入公共预测机制，对无需敏感数据即可预测的词元不产生隐私成本；该方法在处理结构化数据时表现尤为显著。