While massive valuable deep models trained on large-scale data have been released to facilitate the artificial intelligence community, they may encounter attacks in deployment which leads to privacy leakage of training data. In this work, we propose a learning approach termed differentially private data-free distillation (DPDFD) for model conversion that can convert a pretrained model (teacher) into its privacy-preserving counterpart (student) via an intermediate generator without access to training data. The learning collaborates three parties in a unified way. First, massive synthetic data are generated with the generator. Then, they are fed into the teacher and student to compute differentially private gradients by normalizing the gradients and adding noise before performing descent. Finally, the student is updated with these differentially private gradients and the generator is updated by taking the student as a fixed discriminator in an alternate manner. In addition to a privacy-preserving student, the generator can generate synthetic data in a differentially private way for other downstream tasks. We theoretically prove that our approach can guarantee differential privacy and well convergence. Extensive experiments clearly demonstrate that our approach significantly outperform other differentially private generative approaches.

翻译：虽然大量在大规模数据上训练的高价值深度模型已被发布以促进人工智能社区发展，但这些模型在部署过程中可能遭遇攻击，导致训练数据隐私泄露。在本工作中，我们提出一种名为差分隐私无数据蒸馏（DPDFD）的学习方法用于模型转换。该方法通过中间生成器，在无需访问训练数据的情况下，将预训练模型（教师）转换为其隐私保护版本（学生）。该学习过程以统一方式协同三方：首先，使用生成器生成大量合成数据；随后，将这些数据输入教师和学生，通过归一化梯度并在执行梯度下降前添加噪声，计算差分隐私梯度；最后，学生利用这些差分隐私梯度进行更新，而生成器则交替将学生作为固定判别器进行优化。除获得隐私保护学生模型外，生成器还能以差分隐私方式生成合成数据，用于其他下游任务。我们从理论上证明该方法能够保证差分隐私并良好收敛。大量实验明确表明，我们的方法显著优于其他差分隐私生成方法。