Public LLMs such as the Llama 2-Chat have driven huge activity in LLM research. These models underwent alignment training and were considered safe. Recently Qi et al. (2023) reported that even benign fine-tuning (e.g., on seemingly safe datasets) can give rise to unsafe behaviors in the models. The current paper is about methods and best practices to mitigate such loss of alignment. Through extensive experiments on several chat models (Meta's Llama 2-Chat, Mistral AI's Mistral 7B Instruct v0.2, and OpenAI's GPT-3.5 Turbo), this paper uncovers that the prompt templates used during fine-tuning and inference play a crucial role in preserving safety alignment, and proposes the "Pure Tuning, Safe Testing" (PTST) principle -- fine-tune models without a safety prompt, but include it at test time. Fine-tuning experiments on GSM8K, ChatDoctor, and OpenOrca show that PTST significantly reduces the rise of unsafe behaviors, and even almost eliminates them in some cases.

翻译：公开的大语言模型（如Llama 2-Chat）极大地推动了LLM研究领域的活跃度。这些模型经过对齐训练，被认为具有安全性。近期Qi等人（2023）指出，即使是良性的微调（例如在看似安全的数据集上）也可能导致模型产生不安全行为。本文旨在探讨如何缓解这种对齐失效的方法与最佳实践。通过在多个对话模型（Meta的Llama 2-Chat、Mistral AI的Mistral 7B Instruct v0.2及OpenAI的GPT-3.5 Turbo）上的大量实验，本文发现微调和推理阶段使用的提示模板在维持安全对齐方面起着关键作用，并提出了"纯调优、安全测试"（PTST）原则——在微调模型时不使用安全提示，但在测试时加入。针对GSM8K、ChatDoctor和OpenOrca数据集的微调实验表明，PTST能显著降低不安全行为的出现，甚至在某些情况下几乎完全消除这些行为。