Personalized diffusion models (PDMs) have become prominent for adapting pretrained text-to-image models to generate images of specific subjects using minimal training data. However, PDMs are susceptible to minor adversarial perturbations, leading to significant degradation when fine-tuned on corrupted datasets. These vulnerabilities are exploited to create protective perturbations that prevent unauthorized image generation. Existing purification methods attempt to mitigate this issue but often over-purify images, resulting in information loss. In this work, we conduct an in-depth analysis of the fine-tuning process of PDMs through the lens of shortcut learning. We hypothesize and empirically demonstrate that adversarial perturbations induce a latent-space misalignment between images and their text prompts in the CLIP embedding space. This misalignment causes the model to erroneously associate noisy patterns with unique identifiers during fine-tuning, resulting in poor generalization. Based on these insights, we propose a systematic defense framework that includes data purification and contrastive decoupling learning. We first employ off-the-shelf image restoration techniques to realign images with their original semantic meanings in latent space. Then, we introduce contrastive decoupling learning with noise tokens to decouple the learning of personalized concepts from spurious noise patterns. Our study not only uncovers fundamental shortcut learning vulnerabilities in PDMs but also provides a comprehensive evaluation framework for developing stronger protection. Our extensive evaluation demonstrates its superiority over existing purification methods and stronger robustness against adaptive perturbation.

翻译：个性化扩散模型（PDMs）已因其能够利用少量训练数据，将预训练的文本到图像模型适配于生成特定主体图像而变得突出。然而，PDMs容易受到微小对抗性扰动的影响，导致在受污染数据集上进行微调时性能显著下降。这些漏洞被利用来创建保护性扰动，以防止未经授权的图像生成。现有的净化方法试图缓解此问题，但常常过度净化图像，导致信息丢失。在本工作中，我们通过捷径学习的视角，对PDMs的微调过程进行了深入分析。我们假设并通过实验证明，对抗性扰动在CLIP嵌入空间中诱导了图像与其文本提示之间的潜在空间错位。这种错位导致模型在微调过程中错误地将噪声模式与唯一标识符关联起来，从而导致泛化能力差。基于这些见解，我们提出了一个系统性的防御框架，包括数据净化和对比解耦学习。我们首先采用现成的图像恢复技术，使图像在潜在空间中与其原始语义含义重新对齐。然后，我们引入了带有噪声标记的对比解耦学习，以将个性化概念的学习与虚假噪声模式解耦。我们的研究不仅揭示了PDMs中根本性的捷径学习漏洞，还为开发更强的保护提供了全面的评估框架。我们广泛的评估证明了其相对于现有净化方法的优越性以及对自适应扰动更强的鲁棒性。