Larger language models (LLMs) have taken the world by storm with their massive multi-tasking capabilities simply by optimizing over a next-word prediction objective. With the emergence of their properties and encoded knowledge, the risk of LLMs producing harmful outputs increases, making them unfit for scalable deployment for the public. In this work, we propose a new safety evaluation benchmark RED-EVAL that carries out red-teaming. We show that even widely deployed models are susceptible to the Chain of Utterances-based (CoU) prompting, jailbreaking closed source LLM-based systems such as GPT-4 and ChatGPT to unethically respond to more than 65% and 73% of harmful queries. We also demonstrate the consistency of the RED-EVAL across 8 open-source LLMs in generating harmful responses in more than 86% of the red-teaming attempts. Next, we propose RED-INSTRUCT--An approach for the safety alignment of LLMs. It constitutes two phases: 1) HARMFULQA data collection: Leveraging CoU prompting, we collect a dataset that consists of 1.9K harmful questions covering a wide range of topics, 9.5K safe and 7.3K harmful conversations from ChatGPT; 2) SAFE-ALIGN: We demonstrate how the conversational dataset can be used for the safety alignment of LLMs by minimizing the negative log-likelihood over helpful responses and penalizing over harmful responses by gradient accent over sample loss. Our model STARLING, a fine-tuned Vicuna-7B, is observed to be more safely aligned when evaluated on RED-EVAL and HHH benchmarks while preserving the utility of the baseline models (TruthfulQA, MMLU, and BBH).

翻译：大型语言模型（LLMs）通过优化下一词预测目标，凭借其庞大的多任务处理能力席卷全球。随着其特性与编码知识的涌现，LLMs生成有害输出的风险随之增加，使其不适合大规模公开部署。本文提出一种新型安全评估基准RED-EVAL，用于执行红队测试。研究表明，即使广泛部署的模型也易受基于话语链（CoU）的提示攻击，使GPT-4和ChatGPT等闭源LLM系统在超过65%和73%的有害查询中产生不道德回应。我们还在8个开源LLM上验证了RED-EVAL的一致性——在超过86%的红队测试尝试中，模型生成了有害响应。进而提出RED-INSTRUCT——一种用于LLMs安全对齐的方法。该方法包含两个阶段：1）HARMFULQA数据收集：利用CoU提示技术，收集涵盖广泛主题的1.9K个有害问题数据集，以及来自ChatGPT的9.5K个安全对话和7.3K个有害对话；2）SAFE-ALIGN：我们论证如何通过最小化有益响应的负对数似然，并利用样本损失的梯度上升惩罚有害响应，将对话数据集用于LLMs的安全对齐。经评估，基于Vicuna-7B微调的模型STARLING在RED-EVAL和HHH基准测试中展现出更安全的对齐效果，同时保持了基线模型（TruthfulQA、MMLU和BBH）的实用性。