Deep learning techniques have achieved remarkable performance in wide-ranging tasks. However, when trained on privacy-sensitive datasets, the model parameters may expose private information in training data. Prior attempts for differentially private training, although offering rigorous privacy guarantees, lead to much lower model performance than the non-private ones. Besides, different runs of the same training algorithm produce models with large performance variance. To address these issues, we propose DPlis--Differentially Private Learning wIth Smoothing. The core idea of DPlis is to construct a smooth loss function that favors noise-resilient models lying in large flat regions of the loss landscape. We provide theoretical justification for the utility improvements of DPlis. Extensive experiments also demonstrate that DPlis can effectively boost model quality and training stability under a given privacy budget.

翻译：深层学习技术在广泛的任务中取得了显著的成绩。然而,当在对隐私敏感数据集进行培训时,模型参数可能会暴露在培训数据中的私人信息。在进行差别化的私人培训之前,虽然提供了严格的隐私保障,但导致模型性能比非私人培训低得多。此外,同一培训算法的不同运行产生了功能差异很大的模型。为了解决这些问题,我们建议DPlis-差异化私人学习滑动。DPlis的核心思想是构建一个顺畅的损失功能,有利于位于损失分布大片平坦地区的耐噪音模型。我们为DPlis的实用性改进提供了理论上的理由。广泛的实验还表明DPlis可以在特定隐私预算下有效提升模型的质量和培训稳定性。