Larger language models (LLMs) have taken the world by storm with their massive multi-tasking capabilities simply by optimizing over a next-word prediction objective. With the emergence of their properties and encoded knowledge, the risk of LLMs producing harmful outputs increases, making them unfit for scalable deployment for the public. In this work, we propose a new safety evaluation benchmark RED-EVAL that carries out red-teaming. We show that even widely deployed models are susceptible to the Chain of Utterances-based (CoU) prompting, jailbreaking closed source LLM-based systems such as GPT-4 and ChatGPT to unethically respond to more than 65% and 73% of harmful queries. We also demonstrate the consistency of the RED-EVAL across 8 open-source LLMs in generating harmful responses in more than 86% of the red-teaming attempts. Next, we propose RED-INSTRUCT--An approach for the safety alignment of LLMs. It constitutes two phases: 1) HARMFULQA data collection: Leveraging CoU prompting, we collect a dataset that consists of 1.9K harmful questions covering a wide range of topics, 9.5K safe and 7.3K harmful conversations from ChatGPT; 2) SAFE-ALIGN: We demonstrate how the conversational dataset can be used for the safety alignment of LLMs by minimizing the negative log-likelihood over helpful responses and penalizing over harmful responses by gradient accent over sample loss. Our model STARLING, a fine-tuned Vicuna-7B, is observed to be more safely aligned when evaluated on RED-EVAL and HHH benchmarks while preserving the utility of the baseline models (TruthfulQA, MMLU, and BBH).

翻译：大型语言模型（LLMs）通过仅优化下一词预测目标便展现出强大的多任务处理能力，从而席卷全球。随着其属性和编码知识的涌现，LLMs产生有害输出的风险也随之增加，使其无法安全地大规模部署于公共领域。本文提出一项基于红队测试的新型安全评估基准RED-EVAL。研究表明，即使是广泛部署的模型也易受基于话语链（CoU）提示的影响，能够破解GPT-4和ChatGPT等闭源LLM系统，使其对超过65%和73%的有害查询做出不道德回应。我们同时验证了RED-EVAL在8个开源LLM中的一致性，其在超过86%的红队测试尝试中生成了有害回应。接着，我们提出RED-INSTRUCT——一种用于LLM安全对齐的方法，包含两个阶段：1）HARMFULQA数据收集：利用CoU提示收集包含1,900个覆盖广泛主题的有害问题、9,500个安全对话及7,300个有害对话的数据集；2）SAFE-ALIGN：通过最小化有益响应的负对数似然、利用样本损失梯度上升惩罚有害响应，展示如何利用对话数据集实现LLM的安全对齐。经微调Vicuna-7B得到的STARLING模型在RED-EVAL和HHH基准测试中展现出更优的安全对齐性，同时保持了基线模型（TruthfulQA、MMLU和BBH）的实用性。