While deep models have proved successful in learning rich knowledge from massive well-annotated data, they may pose a privacy leakage risk in practical deployment. It is necessary to find an effective trade-off between high utility and strong privacy. In this work, we propose a discriminative-generative distillation approach to learn privacy-preserving deep models. Our key idea is taking models as bridge to distill knowledge from private data and then transfer it to learn a student network via two streams. First, discriminative stream trains a baseline classifier on private data and an ensemble of teachers on multiple disjoint private subsets, respectively. Then, generative stream takes the classifier as a fixed discriminator and trains a generator in a data-free manner. After that, the generator is used to generate massive synthetic data which are further applied to train a variational autoencoder (VAE). Among these synthetic data, a few of them are fed into the teacher ensemble to query labels via differentially private aggregation, while most of them are embedded to the trained VAE for reconstructing synthetic data. Finally, a semi-supervised student learning is performed to simultaneously handle two tasks: knowledge transfer from the teachers with distillation on few privately labeled synthetic data, and knowledge enhancement with tangent-normal adversarial regularization on many triples of reconstructed synthetic data. In this way, our approach can control query cost over private data and mitigate accuracy degradation in a unified manner, leading to a privacy-preserving student model. Extensive experiments and analysis clearly show the effectiveness of the proposed approach.

翻译：尽管深度模型已证明能够从海量标注数据中学习丰富知识，但在实际部署中可能存在隐私泄露风险。因此需要在高效用性与强隐私性之间寻求有效平衡。本文提出一种判别-生成蒸馏方法，用于学习隐私保护的深度模型。核心思想是将模型作为桥梁，从隐私数据中蒸馏知识，随后通过双通道将其迁移至学生网络的学习过程。首先，判别通道分别在隐私数据上训练基线分类器，并在多个互斥隐私子集上训练集成教师模型。随后，生成通道将分类器作为固定判别器，以无数据方式训练生成器。接着，利用该生成器产生大量合成数据，并进一步训练变分自编码器（VAE）。在这些合成数据中，少量数据通过差分隐私聚合机制输入教师集成模型以查询标签，而大部分数据则嵌入训练好的VAE中进行合成数据重构。最后，通过半监督学生学习同时处理两项任务：基于少量隐私标注合成数据的蒸馏知识迁移，以及基于大量重构合成数据三元组的切向-法向对抗正则化知识增强。该方法能以统一方式控制隐私数据查询成本并缓解精度下降，从而获得隐私保护的学生模型。大量实验与分析结果清晰证明了所提方法的有效性。