With the help of conditioning mechanisms, the state-of-the-art diffusion models have achieved tremendous success in guided image generation, particularly in text-to-image synthesis. To gain a better understanding of the training process and potential risks of text-to-image synthesis, we perform a systematic investigation of backdoor attack on text-to-image diffusion models and propose BadT2I, a general multimodal backdoor attack framework that tampers with image synthesis in diverse semantic levels. Specifically, we perform backdoor attacks on three levels of the vision semantics: Pixel-Backdoor, Object-Backdoor and Style-Backdoor. By utilizing a regularization loss, our methods efficiently inject backdoors into a large-scale text-to-image diffusion model while preserving its utility with benign inputs. We conduct empirical experiments on Stable Diffusion, the widely-used text-to-image diffusion model, demonstrating that the large-scale diffusion model can be easily backdoored within a few fine-tuning steps. We conduct additional experiments to explore the impact of different types of textual triggers. Besides, we discuss the backdoor persistence during further training, the findings of which provide insights for the development of backdoor defense methods.

翻译：借助条件化机制，最先进的扩散模型在引导图像生成（尤其是文本到图像合成）中取得了巨大成功。为深入理解文本到图像合成的训练过程及潜在风险，我们系统研究了针对文本到图像扩散模型的后门攻击，并提出BadT2I——一种通用多模态后门攻击框架，能够在不同语义层面篡改图像合成。具体而言，我们在视觉语义的三个层面实施后门攻击：像素级后门、物体级后门和风格级后门。通过利用正则化损失函数，我们的方法能够高效地向大规模文本到图像扩散模型注入后门，同时保持其对良性输入的可用性。我们在广泛使用的文本到图像扩散模型Stable Diffusion上开展实证实验，结果表明大规模扩散模型仅需少量微调步骤即可轻易植入后门。我们进一步设计实验探究不同类型文本触发器的影响。此外，我们讨论后门在持续训练过程中的持久性，相关发现为后门防御方法的发展提供了启示。