Large language models (LLMs) have shown great potential as general-purpose AI assistants in various domains. To meet the requirements of different applications, LLMs are often customized by further fine-tuning. However, the powerful learning ability of LLMs not only enables them to acquire new tasks but also makes them susceptible to learning undesired behaviors. For example, even safety-aligned LLMs can be easily fine-tuned into harmful assistants as the fine-tuning data often contains implicit or explicit harmful content. Can we train LLMs on harmful data without learning harmful behaviors? This paper proposes a controllable training framework that makes harmful behaviors unlearnable during the fine-tuning process. Specifically, we introduce ``security vectors'', a few new parameters that can be separated from the LLM, to ensure LLM's responses are consistent with the harmful behavior. Security vectors are activated during fine-tuning, the consistent behavior makes LLM believe that such behavior has already been learned, there is no need to further optimize for harmful data. During inference, we can deactivate security vectors to restore the LLM's normal behavior. The experimental results show that the security vectors generated by 100 harmful samples are enough to prevent LLM from learning 1000 harmful samples, while preserving the ability to learn other useful information.

翻译：大语言模型（LLM）在多个领域展现出作为通用人工智能助手的巨大潜力。为满足不同应用需求，LLM通常通过进一步微调进行定制。然而，LLM强大的学习能力不仅使其能够获取新任务，还使其容易习得不良行为。例如，即使经过安全对齐的LLM，也可能因微调数据中包含隐式或显式有害内容，被轻易微调成有害助手。我们能否在有害数据上训练LLM而不习得有害行为？本文提出一种可控训练框架，使有害行为在微调过程中变得不可学习。具体而言，我们引入“安全向量”——一组可从LLM中分离的新参数，以确保LLM的响应与有害行为一致。在微调期间激活安全向量，这种一致的行为使LLM相信该行为已被学习，无需针对有害数据进一步优化。在推理阶段，我们可停用安全向量以恢复LLM的正常行为。实验结果表明，仅用100个有害样本生成的安全向量，就足以阻止LLM学习1000个有害样本，同时保留学习其他有用信息的能力。