Diffusion models (DMs) are widely used for generating high-quality high-dimensional images in a non-differentially private manner. To address this challenge, recent papers suggest pre-training DMs with public data, then fine-tuning them with private data using DP-SGD for a relatively short period. In this paper, we further improve the current state of DMs with DP by adopting the Latent Diffusion Models (LDMs). LDMs are equipped with powerful pre-trained autoencoders that map the high-dimensional pixels into lower-dimensional latent representations, in which DMs are trained, yielding a more efficient and fast training of DMs. In our algorithm, DP-LDMs, rather than fine-tuning the entire DMs, we fine-tune only the attention modules of LDMs at varying layers with privacy-sensitive data, reducing the number of trainable parameters by roughly 90% and achieving a better accuracy, compared to fine-tuning the entire DMs. The smaller parameter space to fine-tune with DP-SGD helps our algorithm to achieve new state-of-the-art results in several public-private benchmark data pairs.Our approach also allows us to generate more realistic, high-dimensional images (256x256) and those conditioned on text prompts with differential privacy, which have not been attempted before us, to the best of our knowledge. Our approach provides a promising direction for training more powerful, yet training-efficient differentially private DMs, producing high-quality high-dimensional DP images.

翻译：扩散模型（DMs）被广泛用于以非差分隐私方式生成高质量的高维图像。为应对这一挑战，近期研究建议先使用公开数据预训练扩散模型，再通过DP-SGD在私有数据上对其进行短时间微调。本文通过引入潜在扩散模型（LDMs）进一步改进了当前具有差分隐私特性的扩散模型性能。LDMs配备了强大的预训练自编码器，可将高维像素映射为低维潜在表征，扩散模型在其中进行训练，从而实现更高效快速的训练过程。在我们的算法DP-LDMs中，我们并非对整个扩散模型进行微调，而是仅对LDMs不同层级的注意力模块使用隐私敏感数据进行微调，这使得可训练参数数量减少约90%，同时相比全模型微调实现了更优的准确率。采用DP-SGD微调更小的参数空间，使我们的算法在多个公共-私有基准数据对比中取得了新的最优结果。此外，我们的方法首次实现了差分隐私条件下更逼真高维图像（256×256）及基于文本提示的条件生成——据我们所知，此前尚无此类尝试。该方法为训练更强大且效率更高的差分隐私扩散模型、生成高质量高维差分隐私图像提供了有前景的研究方向。