Denoising diffusion probabilistic models (DDPMs) have been proven capable of synthesizing high-quality images with remarkable diversity when trained on large amounts of data. Typical diffusion models and modern large-scale conditional generative models like text-to-image generative models are vulnerable to overfitting when fine-tuned on extremely limited data. Existing works have explored subject-driven generation using a reference set containing a few images. However, few prior works explore DDPM-based domain-driven generation, which aims to learn the common features of target domains while maintaining diversity. This paper proposes a novel DomainStudio approach to adapt DDPMs pre-trained on large-scale source datasets to target domains using limited data. It is designed to keep the diversity of subjects provided by source domains and get high-quality and diverse adapted samples in target domains. We propose to keep the relative distances between adapted samples to achieve considerable generation diversity. In addition, we further enhance the learning of high-frequency details for better generation quality. Our approach is compatible with both unconditional and conditional diffusion models. This work makes the first attempt to realize unconditional few-shot image generation with diffusion models, achieving better quality and greater diversity than current state-of-the-art GAN-based approaches. Moreover, this work also significantly relieves overfitting for conditional generation and realizes high-quality domain-driven generation, further expanding the applicable scenarios of modern large-scale text-to-image models.

翻译：去噪扩散概率模型（DDPMs）已被证明能够在大量数据上训练时合成具有显著多样性的高质量图像。典型的扩散模型以及现代大规模条件生成模型（如文本到图像生成模型）在极其有限的数据上进行微调时容易出现过拟合。现有研究探索了使用包含少量图像的参考集进行主题驱动生成。然而，很少有前期工作探索基于DDPM的领域驱动生成，该方法旨在学习目标领域的共同特征同时保持多样性。本文提出了一种新颖的DomainStudio方法，利用有限数据将在大规模源数据集上预训练的DDPMs适配到目标领域。该方法旨在保持源领域提供的主题多样性，并在目标领域中获得高质量且多样化的适配样本。我们提出保持适配样本之间的相对距离以实现可观的生成多样性。此外，我们进一步增强了高频细节的学习以提升生成质量。我们的方法兼容无条件扩散模型和条件扩散模型。本文首次尝试利用扩散模型实现无条件的少样本图像生成，与当前最先进的基于GAN的方法相比，实现了更优的质量和更大的多样性。同时，本工作还显著缓解了条件生成中的过拟合问题，并实现了高质量的领域驱动生成，进一步扩展了现代大规模文本到图像模型的应用场景。