Denoising diffusion probabilistic models (DDPMs) have been proven capable of synthesizing high-quality images with remarkable diversity when trained on large amounts of data. Typical diffusion models and modern large-scale conditional generative models like text-to-image generative models are vulnerable to overfitting when fine-tuned on extremely limited data. Existing works have explored subject-driven generation using a reference set containing a few images. However, few prior works explore DDPM-based domain-driven generation, which aims to learn the common features of target domains while maintaining diversity. This paper proposes a novel DomainStudio approach to adapt DDPMs pre-trained on large-scale source datasets to target domains using limited data. It is designed to keep the diversity of subjects provided by source domains and get high-quality and diverse adapted samples in target domains. We propose to keep the relative distances between adapted samples to achieve considerable generation diversity. In addition, we further enhance the learning of high-frequency details for better generation quality. Our approach is compatible with both unconditional and conditional diffusion models. This work makes the first attempt to realize unconditional few-shot image generation with diffusion models, achieving better quality and greater diversity than current state-of-the-art GAN-based approaches. Moreover, this work also significantly relieves overfitting for conditional generation and realizes high-quality domain-driven generation, further expanding the applicable scenarios of modern large-scale text-to-image models.

翻译：去噪扩散概率模型（DDPMs）已被证明在大量数据训练下能够合成具有显著多样性的高质量图像。典型的扩散模型及当代大规模条件生成模型（如文本到图像生成模型）在极有限数据上微调时容易过拟合。现有工作已探索利用包含少量图像的参考集实现主体驱动生成，但鲜有研究探讨基于DDPMs的领域驱动生成——即在保持多样性的同时学习目标域共同特征。本文提出新型DomainStudio方法，使在大规模源数据集上预训练的DDPMs能通过有限数据适配目标域。该方法旨在保留源域提供的主题多样性，同时获取目标域中高质量且多样化的适配样本。我们提出保持适配样本间的相对距离以实现可观的生成多样性，并进一步强化高频细节学习以提升生成质量。本方法兼容无条件与条件扩散模型。该工作首次尝试利用扩散模型实现无条件的少样本图像生成，在质量和多样性上均超越当前基于GAN的最先进方法。此外，本研究还显著缓解了条件生成中的过拟合现象，实现了高质量领域驱动生成，进一步拓展了现代大规模文本到图像模型的适用场景。