Instruction tuning is an effective technique to align large language models (LLMs) with human intents. In this work, we investigate how an adversary can exploit instruction tuning by injecting specific instruction-following examples into the training data that intentionally changes the model's behavior. For example, an adversary can achieve content injection by injecting training examples that mention target content and eliciting such behavior from downstream models. To achieve this goal, we propose \textit{AutoPoison}, an automated data poisoning pipeline. It naturally and coherently incorporates versatile attack goals into poisoned data with the help of an oracle LLM. We showcase two example attacks: content injection and over-refusal attacks, each aiming to induce a specific exploitable behavior. We quantify and benchmark the strength and the stealthiness of our data poisoning scheme. Our results show that AutoPoison allows an adversary to change a model's behavior by poisoning only a small fraction of data while maintaining a high level of stealthiness in the poisoned examples. We hope our work sheds light on how data quality affects the behavior of instruction-tuned models and raises awareness of the importance of data quality for responsible deployments of LLMs. Code is available at \url{https://github.com/azshue/AutoPoison}.

翻译：指令调优是使大语言模型（LLMs）与人类意图对齐的有效技术。本文研究对手如何通过将特定指令遵循示例注入训练数据来利用指令微调技术，从而有意改变模型行为。例如，对手可以通过注入包含目标内容的训练示例来诱导下游模型产生此类行为，实现内容注入攻击。为此，我们提出自动数据投毒流水线AutoPoison。该方法借助预言机LLM，能够自然连贯地将多种攻击目标注入中毒数据。我们展示了两种示例攻击：内容注入攻击和过度拒绝攻击，分别旨在诱导特定的可利用行为。我们量化并基准测试了数据投毒方案的强度和隐蔽性。实验结果表明，AutoPoison允许对手仅污染少量数据即可改变模型行为，同时保持中毒示例的高度隐蔽性。我们期望这项工作能够揭示数据质量如何影响指令调优模型的行为，并提高对LLM负责任部署中数据质量重要性的认识。代码开源于\url{https://github.com/azshue/AutoPoison}。