While massive valuable deep models trained on large-scale data have been released to facilitate the artificial intelligence community, they may encounter attacks in deployment which leads to privacy leakage of training data. In this work, we propose a learning approach termed differentially private data-free distillation (DPDFD) for model conversion that can convert a pretrained model (teacher) into its privacy-preserving counterpart (student) via an intermediate generator without access to training data. The learning collaborates three parties in a unified way. First, massive synthetic data are generated with the generator. Then, they are fed into the teacher and student to compute differentially private gradients by normalizing the gradients and adding noise before performing descent. Finally, the student is updated with these differentially private gradients and the generator is updated by taking the student as a fixed discriminator in an alternate manner. In addition to a privacy-preserving student, the generator can generate synthetic data in a differentially private way for other downstream tasks. We theoretically prove that our approach can guarantee differential privacy and well convergence. Extensive experiments clearly demonstrate that our approach significantly outperform other differentially private generative approaches.

翻译：尽管大量基于大规模数据训练的深度模型已被发布以促进人工智能社区发展，但在部署过程中可能遭遇攻击，导致训练数据隐私泄露。本文提出一种名为差分隐私无数据蒸馏（DPDFD）的学习方法用于模型转换，该方法通过中间生成器将预训练模型（教师模型）转换为具有隐私保护能力的对应模型（学生模型），且无需访问原始训练数据。该学习过程以统一方式协作三方：首先，通过生成器生成海量合成数据；其次，将这些数据分别输入教师模型与学生模型，通过归一化梯度并在参数更新前添加噪声来计算差分隐私梯度；最后，学生模型利用这些差分隐私梯度进行更新，而生成器则以学生模型为固定判别器交替更新。除生成隐私保护的学生模型外，该生成器还能以差分隐私方式生成合成数据，用于其他下游任务。我们从理论上证明该方法能保证差分隐私并具有良好的收敛性。大量实验清晰表明，本方法显著优于其他差分隐私生成式方法。