Fine-tuning Stable Diffusion enables subject-driven image synthesis by adapting the model to generate images containing specific subjects. However, existing fine-tuning methods suffer from two key issues: underfitting, where the model fails to reliably capture subject identity, and overfitting, where it memorizes the subject image and reduces background diversity. To address these challenges, we propose two auxiliary consistency losses for diffusion fine-tuning. First, a prior consistency regularization loss ensures that the predicted diffusion noise for prior (non-subject) images remains consistent with that of the pretrained model, improving fidelity. Second, a subject consistency regularization loss enhances the fine-tuned model's robustness to multiplicative noise modulated latent code, helping to preserve subject identity while improving diversity. Our experimental results demonstrate that incorporating these losses into fine-tuning not only preserves subject identity but also enhances image diversity, outperforming DreamBooth in terms of CLIP scores, background variation, and overall visual quality.

翻译：通过对Stable Diffusion进行微调，可以使模型适应生成包含特定主体的图像，从而实现主体驱动的图像合成。然而，现有的微调方法存在两个关键问题：欠拟合，即模型无法可靠地捕捉主体身份；以及过拟合，即模型记住了主体图像并降低了背景多样性。为了解决这些挑战，我们提出了两种用于扩散模型微调的辅助一致性损失。首先，先验一致性正则化损失确保对于先验（非主体）图像预测的扩散噪声与预训练模型保持一致，从而提高保真度。其次，主体一致性正则化损失增强了微调模型对由乘性噪声调制的潜在代码的鲁棒性，有助于在保持主体身份的同时提高多样性。我们的实验结果表明，将这些损失纳入微调过程不仅能保持主体身份，还能增强图像多样性，在CLIP分数、背景变化和整体视觉质量方面均优于DreamBooth。