With the help of conditioning mechanisms, the state-of-the-art diffusion models have achieved tremendous success in guided image generation, particularly in text-to-image synthesis. To gain a better understanding of the training process and potential risks of text-to-image synthesis, we perform a systematic investigation of backdoor attack on text-to-image diffusion models and propose BadT2I, a general multimodal backdoor attack framework that tampers with image synthesis in diverse semantic levels. Specifically, we perform backdoor attacks on three levels of the vision semantics: Pixel-Backdoor, Object-Backdoor and Style-Backdoor. By utilizing a regularization loss, our methods efficiently inject backdoors into a large-scale text-to-image diffusion model while preserving its utility with benign inputs. We conduct empirical experiments on Stable Diffusion, the widely-used text-to-image diffusion model, demonstrating that the large-scale diffusion model can be easily backdoored within a few fine-tuning steps. We conduct additional experiments to explore the impact of different types of textual triggers, as well as the backdoor persistence during further training, providing insights for the development of backdoor defense methods. Besides, our investigation may contribute to the copyright protection of text-to-image models in the future.

翻译：借助条件机制，最先进的扩散模型在引导图像生成（尤其是文本到图像合成）领域取得了巨大成功。为更深入理解文本到图像合成的训练过程及其潜在风险，我们对文本到图像扩散模型的后门攻击进行了系统性研究，并提出BadT2I——一种能在多语义层面篡改图像合成的通用多模态后门攻击框架。具体而言，我们从视觉语义的三个层面实施后门攻击：像素级后门、对象级后门和风格级后门。通过引入正则化损失，我们的方法能在保持模型对良性输入可用性的同时，高效地将后门注入大规模文本到图像扩散模型。针对广泛使用的文本到图像扩散模型Stable Diffusion的实验表明，仅需少量微调步骤即可轻松植入后门。我们进一步探索了不同文本触发器类型的影响及后门在后续训练中的持久性，为后门防御方法的发展提供了启示。此外，本项研究未来可能为文本到图像模型的版权保护做出贡献。