Data poisoning attacks manipulate training data to introduce unexpected behaviors into machine learning models at training time. For text-to-image generative models with massive training datasets, current understanding of poisoning attacks suggests that a successful attack would require injecting millions of poison samples into their training pipeline. In this paper, we show that poisoning attacks can be successful on generative models. We observe that training data per concept can be quite limited in these models, making them vulnerable to prompt-specific poisoning attacks, which target a model's ability to respond to individual prompts. We introduce Nightshade, an optimized prompt-specific poisoning attack where poison samples look visually identical to benign images with matching text prompts. Nightshade poison samples are also optimized for potency and can corrupt an Stable Diffusion SDXL prompt in <100 poison samples. Nightshade poison effects "bleed through" to related concepts, and multiple attacks can composed together in a single prompt. Surprisingly, we show that a moderate number of Nightshade attacks can destabilize general features in a text-to-image generative model, effectively disabling its ability to generate meaningful images. Finally, we propose the use of Nightshade and similar tools as a last defense for content creators against web scrapers that ignore opt-out/do-not-crawl directives, and discuss possible implications for model trainers and content creators.

翻译：数据投毒攻击通过在训练时操纵训练数据，使机器学习模型产生意外行为。对于具有海量训练数据集的文本到图像生成模型，当前对投毒攻击的理解认为，成功攻击需向训练流程注入数百万个中毒样本。本文表明，投毒攻击对生成模型亦可奏效。我们观察到，这些模型中每个概念的训练数据可能相当有限，使其容易受到针对模型响应特定提示能力的提示特定投毒攻击。我们提出Nightshade，一种优化的提示特定投毒攻击方法，其中中毒样本在视觉上与匹配文本提示的良性图像完全相同。Nightshade中毒样本还针对效能进行了优化，可在少于100个中毒样本的条件下破坏Stable Diffusion SDXL提示。Nightshade的中毒效果会"渗透"至相关概念，且多个攻击可组合至单一提示中。令人惊讶的是，适度的Nightshade攻击即可破坏文本到图像生成模型的通用特征，有效使其丧失生成有意义图像的能力。最后，我们建议将Nightshade及类似工具作为内容创作者对抗忽略退出/禁止爬取指令的网络爬虫的最后防线，并讨论了对模型训练者和内容创作者的潜在影响。