Diffusion models have proven to be highly effective in generating high-quality images. However, adapting large pre-trained diffusion models to new domains remains an open challenge, which is critical for real-world applications. This paper proposes DiffFit, a parameter-efficient strategy to fine-tune large pre-trained diffusion models that enable fast adaptation to new domains. DiffFit is embarrassingly simple that only fine-tunes the bias term and newly-added scaling factors in specific layers, yet resulting in significant training speed-up and reduced model storage costs. Compared with full fine-tuning, DiffFit achieves 2$\times$ training speed-up and only needs to store approximately 0.12\% of the total model parameters. Intuitive theoretical analysis has been provided to justify the efficacy of scaling factors on fast adaptation. On 8 downstream datasets, DiffFit achieves superior or competitive performances compared to the full fine-tuning while being more efficient. Remarkably, we show that DiffFit can adapt a pre-trained low-resolution generative model to a high-resolution one by adding minimal cost. Among diffusion-based methods, DiffFit sets a new state-of-the-art FID of 3.02 on ImageNet 512$\times$512 benchmark by fine-tuning only 25 epochs from a public pre-trained ImageNet 256$\times$256 checkpoint while being 30$\times$ more training efficient than the closest competitor.

翻译：扩散模型在生成高质量图像方面已被证明非常有效。然而，将大型预训练扩散模型适配到新领域仍是一项开放挑战，这对实际应用至关重要。本文提出DiffFit，一种参数高效的策略，用于微调大型预训练扩散模型，使其能快速适配新领域。DiffFit极为简单，仅微调偏置项和在特定层新增的缩放因子，却能显著加速训练并降低模型存储成本。与全量微调相比，DiffFit实现了2倍的训练加速，且仅需存储约0.12%的模型总参数。我们提供了直观的理论分析，以证明缩放因子在快速适配中的有效性。在8个下游数据集上，DiffFit在优于或媲美全量微调性能的同时，更具高效性。值得注意的是，我们表明DiffFit能以极低代价将预训练的低分辨率生成模型适配为高分辨率模型。在基于扩散的方法中，DiffFit通过在公开预训练的ImageNet 256×256检查点上仅微调25个周期，在ImageNet 512×512基准测试上取得了3.02的最先进FID分数，同时训练效率比最接近的竞争者高30倍。