Differentially private training algorithms like DP-SGD protect sensitive training data by ensuring that trained models do not reveal private information. An alternative approach, which this paper studies, is to use a sensitive dataset to generate synthetic data that is differentially private with respect to the original data, and then non-privately training a model on the synthetic data. Doing so has several advantages: synthetic data can be reused for other tasks (including for hyper parameter tuning), retained indefinitely, and shared with third parties without sacrificing privacy. However, generating private synthetic data is much harder than training a private model. To improve performance on text data, recent work has utilized public data by starting with a pre-trained generative language model and privately fine-tuning it on sensitive data. This model can be used to sample a DP synthetic dataset. While this strategy seems straightforward, executing it has proven problematic. Previous approaches either show significant performance loss, or have, as we show, critical design flaws. In this paper we demonstrate that a proper training objective along with tuning fewer parameters results in excellent DP synthetic data quality. Our approach is competitive with direct DP-training of downstream classifiers in terms of performance on downstream tasks. Further, we demonstrate that our DP synthetic data is not only useful for downstream classifier training, but also to tune those same models.

翻译：差分隐私训练算法（如DP-SGD）通过确保训练后的模型不泄露私有信息来保护敏感训练数据。本文研究的另一种方法是：使用敏感数据集生成与原始数据满足差分隐私的合成数据，然后基于该合成数据非隐私地训练模型。这种方法具有多项优势：合成数据可重复用于其他任务（包括超参数调优）、可无限期保留，以及与第三方共享而无需牺牲隐私。然而，生成私有合成数据比训练私有模型困难得多。为提升文本数据的性能，近期研究通过利用公开数据：从预训练的生成式语言模型出发，在敏感数据上进行隐私微调，进而采样生成差分隐私合成数据集。尽管这一策略看似直接，实际执行却面临挑战。现有方法要么在性能上显著下降，要么如我们所示存在关键设计缺陷。本文证明：采用恰当的训练目标并减少调优参数数量，能够获得质量优异的差分隐私合成数据。我们的方法在下游任务性能上可与直接进行差分隐私训练的下游分类器相媲美。此外，我们验证了所生成的差分隐私合成数据不仅适用于下游分类器训练，还可用于这类模型的超参数调优。