As large language models (LLMs) become increasingly prevalent across many real-world applications, understanding and enhancing their robustness to adversarial attacks is of paramount importance. Existing methods for identifying adversarial prompts tend to focus on specific domains, lack diversity, or require extensive human annotations. To address these limitations, we present Rainbow Teaming, a novel black-box approach for producing a diverse collection of adversarial prompts. Rainbow Teaming casts adversarial prompt generation as a quality-diversity problem and uses open-ended search to generate prompts that are both effective and diverse. Focusing on the safety domain, we use Rainbow Teaming to target various state-of-the-art LLMs, including the Llama 2 and Llama 3 models. Our approach reveals hundreds of effective adversarial prompts, with an attack success rate exceeding 90% across all tested models. Furthermore, we demonstrate that prompts generated by Rainbow Teaming are highly transferable and that fine-tuning models with synthetic data generated by our method significantly enhances their safety without sacrificing general performance or helpfulness. We additionally explore the versatility of Rainbow Teaming by applying it to question answering and cybersecurity, showcasing its potential to drive robust open-ended self-improvement in a wide range of applications.

翻译：随着大语言模型（LLM）在众多实际应用中的日益普及，理解并增强其对抗攻击的鲁棒性至关重要。现有的识别对抗性提示的方法往往局限于特定领域、缺乏多样性，或需要大量人工标注。为应对这些局限性，我们提出了彩虹组队，一种新颖的黑盒方法，用于生成多样化的对抗性提示集合。彩虹组队将对抗性提示生成视为一个质量-多样性问题，并利用开放式搜索来生成既有效又多样化的提示。聚焦于安全领域，我们使用彩虹组队针对包括Llama 2和Llama 3模型在内的多种先进LLM进行测试。我们的方法揭示了数百个有效的对抗性提示，在所有测试模型上的攻击成功率超过90%。此外，我们证明彩虹组队生成的提示具有高度可迁移性，并且使用我们方法生成的合成数据对模型进行微调，能在不牺牲通用性能或帮助性的前提下显著提升其安全性。我们还通过将彩虹组队应用于问答和网络安全领域，探索了其多功能性，展示了其在广泛应用中推动鲁棒的开放式自我改进的潜力。