Differentially private (DP) training methods like DP-SGD can protect sensitive training data by ensuring that ML models will not reveal private information. An alternative approach, which this paper studies, is to use a sensitive dataset to generate a new synthetic dataset which is differentially private with respect to the original data. Doing so has several advantages: synthetic data can be reused for other tasks (including for hyper parameter tuning), retained indefinitely, or shared with third parties without sacrificing privacy. However, obtaining DP data is much harder than introducing DP during training. To make it feasible for text, recent work has utilized public data by starting with a pre-trained generative language model and privately finetuning it on sensitive data. This model can be used to sample a DP synthetic dataset. While this strategy seems straightforward, executing it has proven problematic. Previous approaches either show significant performance loss, or have, as we show, critical design flaws. In this paper we demonstrate that a proper training objective along with tuning fewer parameters results in excellent DP synthetic data quality. Our approach is competitive with direct DP-training of downstream classifiers in terms of performance on downstream tasks. We also demonstrate that our DP synthetic data is not only useful for downstream classifier training, but also to tune those same models.

翻译：差分隐私（DP）训练方法（如DP-SGD）通过确保机器学习模型不会泄露隐私信息来保护敏感训练数据。本文研究另一种替代方法：使用敏感数据集生成新的合成数据集，使其在差分隐私意义上相对于原始数据保持私密性。这样做具有若干优势：合成数据可重复用于其他任务（包括超参数调优）、无限期保留，或在不牺牲隐私的前提下与第三方共享。然而，获取差分隐私数据远比在训练过程中引入DP更难。为了使其在文本领域可行，近期工作利用公开数据，从预训练的生成式语言模型出发，在敏感数据上进行私有微调。该模型可用于采样合成差分隐私数据集。尽管这一策略看似直接，但实际执行中却问题重重。先前的方法要么表现出显著的性能损失，要么（如我们所揭示）存在关键的设计缺陷。本文证明，合理的训练目标配合较少的参数调优，能够实现卓越的差分隐私合成数据质量。我们的方法在下游任务性能上与直接对下游分类器进行差分隐私训练具有竞争力。我们还证明，我们的差分隐私合成数据不仅有助于下游分类器训练，还可用于对这些模型本身进行调优。