Diffusion models (DMs) are widely used for generating high-quality high-dimensional images in a non-differentially private manner. To address this challenge, recent papers suggest pre-training DMs with public data, then fine-tuning them with private data using DP-SGD for a relatively short period. In this paper, we further improve the current state of DMs with DP by adopting the Latent Diffusion Models (LDMs). LDMs are equipped with powerful pre-trained autoencoders that map the high-dimensional pixels into lower-dimensional latent representations, in which DMs are trained, yielding a more efficient and fast training of DMs. In our algorithm, DP-LDMs, rather than fine-tuning the entire DMs, we fine-tune only the attention modules of LDMs at varying layers with privacy-sensitive data, reducing the number of trainable parameters by roughly 90% and achieving a better accuracy, compared to fine-tuning the entire DMs. The smaller parameter space to fine-tune with DP-SGD helps our algorithm to achieve new state-of-the-art results in several public-private benchmark data pairs.Our approach also allows us to generate more realistic, high-dimensional images (256x256) and those conditioned on text prompts with differential privacy, which have not been attempted before us, to the best of our knowledge. Our approach provides a promising direction for training more powerful, yet training-efficient differentially private DMs, producing high-quality high-dimensional DP images.

翻译：扩散模型被广泛用于以非差分隐私方式生成高质量的高维图像。针对这一挑战，近期研究提出先用公开数据预训练扩散模型，再通过DP-SGD在私有数据上对其进行短时间微调。本文通过采用潜在扩散模型（LDMs）进一步改进了当前带差分隐私的扩散模型性能。LDMs配备了强大的预训练自编码器，可将高维像素映射至低维潜在表征空间，并在该空间中训练扩散模型，从而实现更高效快速的训练。在我们的算法DP-LDMs中，我们并非微调整个扩散模型，而是仅对LDMs不同层的注意力模块使用隐私敏感数据进行微调，使可训练参数数量减少约90%，且相比微调整个模型获得了更高准确率。由于使用DP-SGD微调的参数空间更小，该算法在多个公开-私有基准数据对中取得了新的最佳结果。我们的方法还能生成更真实的高维图像（256×256）以及基于文本提示的差分隐私图像——据我们所知，此前尚未有研究尝试过此类工作。本方法为训练更强大且训练高效的差分隐私扩散模型提供了有前景的方向，可生成高质量高维的差分隐私图像。