Large language models (LLMs) are susceptible to red teaming attacks, which can induce LLMs to generate harmful content. Previous research constructs attack prompts via manual or automatic methods, which have their own limitations on construction cost and quality. To address these issues, we propose an integrated approach that combines manual and automatic methods to economically generate high-quality attack prompts. Specifically, considering the impressive capabilities of newly emerged LLMs, we propose an attack framework to instruct LLMs to mimic human-generated prompts through in-context learning. Furthermore, we propose a defense framework that fine-tunes victim LLMs through iterative interactions with the attack framework to enhance their safety against red teaming attacks. Extensive experiments on different LLMs validate the effectiveness of our proposed attack and defense frameworks. Additionally, we release a series of attack prompts datasets named SAP with varying sizes, facilitating the safety evaluation and enhancement of more LLMs. Our code and dataset is available on https://github.com/Aatrox103/SAP .

翻译：大型语言模型（LLMs）易受红队攻击，此类攻击可诱导LLMs生成有害内容。现有研究通过人工或自动方法构建攻击提示，但在构建成本与质量上各有局限。为解决这些问题，我们提出一种结合人工与自动方法的集成方案，以经济高效地生成高质量攻击提示。具体而言，考虑到新兴LLMs的强大能力，我们设计了一个攻击框架，通过上下文学习指导LLMs模拟人类生成的提示。此外，我们提出防御框架，通过攻击框架的迭代交互对受害者LLMs进行微调，以增强其抵御红队攻击的安全性。在不同LLMs上的大量实验验证了我们提出的攻击与防御框架的有效性。同时，我们发布了名为SAP的系列不同规模的攻击提示数据集，以促进更多LLMs的安全性评估与增强。我们的代码与数据集发布于 https://github.com/Aatrox103/SAP 。