Diffusion models have been remarkably successful in data synthesis. Such successes have also driven diffusion models to apply to sensitive data, such as human face data, but this might bring about severe privacy concerns. In this work, we systematically present the first privacy study about property inference attacks against diffusion models, in which adversaries aim to extract sensitive global properties of the training set from a diffusion model, such as the proportion of the training data for certain sensitive properties. Specifically, we consider the most practical attack scenario: adversaries are only allowed to obtain synthetic data. Under this realistic scenario, we evaluate the property inference attacks on different types of samplers and diffusion models. A broad range of evaluations shows that various diffusion models and their samplers are all vulnerable to property inference attacks. Furthermore, one case study on off-the-shelf pre-trained diffusion models also demonstrates the effectiveness of the attack in practice. Finally, we propose a new model-agnostic plug-in method PriSampler to mitigate the property inference of diffusion models. PriSampler can be directly applied to well-trained diffusion models and support both stochastic and deterministic sampling. Extensive experiments illustrate the effectiveness of our defense and it makes adversaries infer the proportion of properties as close as random guesses. PriSampler also shows its significantly superior performance to diffusion models trained with differential privacy on both model utility and defense performance.

翻译：扩散模型在数据合成方面取得了显著成功。这些成功也促使扩散模型被应用于敏感数据（如人脸数据），但这可能引发严重的隐私问题。本文首次系统性地研究了针对扩散模型的属性推断攻击隐私问题——攻击者旨在从扩散模型中提取训练集的敏感全局属性，例如训练数据中特定敏感属性的占比。具体而言，我们考虑了最实际的攻击场景：攻击者仅允许获取合成数据。在这一现实场景下，我们评估了针对不同类型采样器和扩散模型的属性推断攻击。广泛评估表明，各类扩散模型及其采样器均易受属性推断攻击。此外，针对现成预训练扩散模型的案例研究也验证了攻击在实际中的有效性。最后，我们提出了一种全新的模型无关插件式方法PriSampler，用于缓解扩散模型的属性推断问题。PriSampler可直接应用于已训练好的扩散模型，并支持随机采样和确定性采样。大量实验证明了我们防御方法的有效性，可使攻击者对属性比例的推断结果接近随机猜测。与采用差分隐私训练的扩散模型相比，PriSampler在模型效用和防御性能上均表现出显著优越性。