Latent diffusion models achieve state-of-the-art performance on a variety of generative tasks, such as image synthesis and image editing. However, the robustness of latent diffusion models is not well studied. Previous works only focus on the adversarial attacks against the encoder or the output image under white-box settings, regardless of the denoising process. Therefore, in this paper, we aim to analyze the robustness of latent diffusion models more thoroughly. We first study the influence of the components inside latent diffusion models on their white-box robustness. In addition to white-box scenarios, we evaluate the black-box robustness of latent diffusion models via transfer attacks, where we consider both prompt-transfer and model-transfer settings and possible defense mechanisms. However, all these explorations need a comprehensive benchmark dataset, which is missing in the literature. Therefore, to facilitate the research of the robustness of latent diffusion models, we propose two automatic dataset construction pipelines for two kinds of image editing models and release the whole dataset. Our code and dataset are available at \url{https://github.com/jpzhang1810/LDM-Robustness}.
翻译:潜在扩散模型在图像合成和图像编辑等多种生成任务中取得了最先进的性能。然而,潜在扩散模型的鲁棒性尚未得到充分研究。先前的工作仅关注白盒设置下针对编码器或输出图像的对抗攻击,而忽略了去噪过程。因此,本文旨在更全面地分析潜在扩散模型的鲁棒性。我们首先研究了潜在扩散模型内部组件对其白盒鲁棒性的影响。除白盒场景外,我们还通过迁移攻击评估了潜在扩散模型的黑盒鲁棒性,其中考虑了提示迁移和模型迁移设置以及可能的防御机制。然而,所有这些探索都需要一个全面的基准数据集,而现有文献中缺少这一数据集。因此,为促进潜在扩散模型鲁棒性的研究,我们针对两种图像编辑模型提出了两种自动数据集构建流程,并发布了完整数据集。我们的代码和数据集可在 \url{https://github.com/jpzhang1810/LDM-Robustness} 获取。