Data poisoning attacks manipulate training data to introduce unexpected behaviors into machine learning models at training time. For text-to-image generative models with massive training datasets, current understanding of poisoning attacks suggests that a successful attack would require injecting millions of poison samples into their training pipeline. In this paper, we show that poisoning attacks can be successful on generative models. We observe that training data per concept can be quite limited in these models, making them vulnerable to prompt-specific poisoning attacks, which target a model's ability to respond to individual prompts. We introduce Nightshade, an optimized prompt-specific poisoning attack where poison samples look visually identical to benign images with matching text prompts. Nightshade poison samples are also optimized for potency and can corrupt an Stable Diffusion SDXL prompt in <100 poison samples. Nightshade poison effects "bleed through" to related concepts, and multiple attacks can composed together in a single prompt. Surprisingly, we show that a moderate number of Nightshade attacks can destabilize general features in a text-to-image generative model, effectively disabling its ability to generate meaningful images. Finally, we propose the use of Nightshade` and similar tools as a last defense for content creators against web scrapers that ignore opt-out/do-not-crawl directives, and discuss possible implications for model trainers and content creators.

翻译：数据投毒攻击通过在训练时操控训练数据，使机器学习模型产生预期外的行为。对于使用海量训练数据的文本到图像生成模型而言，当前的投毒攻击认知表明，成功攻击需要向训练流程注入数百万个投毒样本。本文证明投毒攻击对生成模型同样有效。我们观察到，这些模型中每个概念的训练数据可能极为有限，这使得它们容易受到特定提示词下的投毒攻击——此类攻击针对模型响应单个提示词的能力。我们提出Nightshade，一种优化的特定提示词投毒攻击方法，其投毒样本在视觉上与匹配文本提示词的良性图像完全一致。Nightshade投毒样本还针对效力进行了优化，可在不到100个投毒样本的情况下破坏Stable Diffusion SDXL的提示词生成效果。Nightshade的投毒效应会"渗透"至相关概念，且多种攻击可组合在同一提示词中。令人惊讶的是，我们发现中等数量的Nightshade攻击就能破坏文本到图像生成模型的通用特征，使其基本丧失生成有意义图像的能力。最后，我们将Nightshade及类似工具作为内容创作者针对无视拒绝爬取/禁止爬取指令的网络爬虫的最后防线，并探讨其对模型训练者和内容创作者的潜在影响。