Knowledge distillation in neural networks refers to compressing a large model or dataset into a smaller version of itself. We introduce Privacy Distillation, a framework that allows a text-to-image generative model to teach another model without exposing it to identifiable data. Here, we are interested in the privacy issue faced by a data provider who wishes to share their data via a multimodal generative model. A question that immediately arises is ``How can a data provider ensure that the generative model is not leaking identifiable information about a patient?''. Our solution consists of (1) training a first diffusion model on real data (2) generating a synthetic dataset using this model and filtering it to exclude images with a re-identifiability risk (3) training a second diffusion model on the filtered synthetic data only. We showcase that datasets sampled from models trained with privacy distillation can effectively reduce re-identification risk whilst maintaining downstream performance.

翻译：神经网络中的知识蒸馏是指将大型模型或数据集压缩为更小版本的技术。我们提出“隐私蒸馏”框架，该框架允许文本到图像的生成模型在不暴露可识别数据的情况下教导另一个模型。本文关注数据提供者希望通过多模态生成模型共享数据时面临的隐私问题。由此衍生出一个关键问题：“数据提供者如何确保生成模型不会泄露患者可识别信息？”我们的解决方案包括：(1) 在真实数据上训练第一个扩散模型；(2) 利用该模型生成合成数据集，并过滤掉存在重识别风险的图像；(3) 仅在过滤后的合成数据上训练第二个扩散模型。实验表明，采用隐私蒸馏训练的模型所采样的数据集能够在保持下游任务性能的同时，有效降低重识别风险。