Diffusion models (DMs) are widely used for generating high-quality image datasets. However, since they operate directly in the high-dimensional pixel space, optimization of DMs is computationally expensive, requiring long training times. This contributes to large amounts of noise being injected into the differentially private learning process, due to the composability property of differential privacy. To address this challenge, we propose training Latent Diffusion Models (LDMs) with differential privacy. LDMs use powerful pre-trained autoencoders to reduce the high-dimensional pixel space to a much lower-dimensional latent space, making training DMs more efficient and fast. Unlike [Ghalebikesabi et al., 2023] that pre-trains DMs with public data then fine-tunes them with private data, we fine-tune only the attention modules of LDMs at varying layers with privacy-sensitive data, reducing the number of trainable parameters by approximately 96% compared to fine-tuning the entire DM. We test our algorithm on several public-private data pairs, such as ImageNet as public data and CIFAR10 and CelebA as private data, and SVHN as public data and MNIST as private data. Our approach provides a promising direction for training more powerful, yet training-efficient differentially private DMs that can produce high-quality synthetic images.

翻译：扩散模型（DMs）被广泛用于生成高质量图像数据集。然而，由于这些模型直接在高维像素空间中操作，其优化过程计算代价高昂，需要较长的训练时间。由于差分隐私的组合性质，这会导致在差分隐私学习过程中注入大量噪声。为解决这一挑战，我们提出使用差分隐私训练潜在扩散模型（LDMs）。LDMs利用强大的预训练自编码器将高维像素空间压缩至更低维度的潜在空间，使得训练DMs更加高效快速。与[Ghalebikesabi等人，2023]中使用公开数据预训练DMs再使用私有数据微调的方法不同，我们仅对LDMs中不同层级的注意力模块使用隐私敏感数据进行微调，与微调整个DM相比，可训练参数量减少了约96%。我们在多组公私数据对（如以ImageNet为公开数据、CIFAR10和CelebA为私有数据，以及以SVHN为公开数据、MNIST为私有数据）上测试了算法。该方法为训练更强大且训练高效的差分隐私DMs提供了有前景的方向，可生成高质量的合成图像。