Diffusion models have been remarkably successful in data synthesis. However, when these models are applied to sensitive datasets, such as banking and human face data, they might bring up severe privacy concerns. This work systematically presents the first privacy study about property inference attacks against diffusion models, where adversaries aim to extract sensitive global properties of its training set from a diffusion model. Specifically, we focus on the most practical attack scenario: adversaries are restricted to accessing only synthetic data. Under this realistic scenario, we conduct a comprehensive evaluation of property inference attacks on various diffusion models trained on diverse data types, including tabular and image datasets. A broad range of evaluations reveals that diffusion models and their samplers are universally vulnerable to property inference attacks. In response, we propose a new model-agnostic plug-in method PriSampler to mitigate the risks of the property inference of diffusion models. PriSampler can be directly applied to well-trained diffusion models and support both stochastic and deterministic sampling. Extensive experiments illustrate the effectiveness of our defense, and it can lead adversaries to infer the proportion of properties as close as predefined values that model owners wish. Notably, PriSampler also shows its significantly superior performance to diffusion models trained with differential privacy on both model utility and defense performance. This work will elevate the awareness of preventing property inference attacks and encourage privacy-preserving synthetic data release.

翻译：扩散模型在数据合成领域取得了显著成功。然而，当将这些模型应用于银行、人脸数据等敏感数据集时，可能引发严重的隐私问题。本文首次系统性地开展了针对扩散模型的属性推断攻击的隐私研究，攻击者旨在从扩散模型中提取其训练集的敏感全局属性。具体而言，我们聚焦于最实际的攻击场景：攻击者仅能访问合成数据。在此现实场景下，我们对基于多种数据类型（包括表格数据和图像数据）训练的各类扩散模型进行了全面的属性推断攻击评估。广泛评估表明，扩散模型及其采样器普遍易受属性推断攻击。为此，我们提出了一种新型模型无关的即插即用方法PriSampler，以减轻扩散模型属性推断的风险。PriSampler可直接应用于预训练的扩散模型，且支持随机采样与确定性采样。大量实验证明了我们防御方法的有效性，它能够迫使攻击者推断出的属性比例趋近于模型所有者预设的目标值。值得注意的是，在模型效用与防御性能方面，PriSampler均显著优于采用差分隐私训练的扩散模型。本研究将提升对防御属性推断攻击的重视，并促进隐私保护的合成数据发布。