The rapid advancement of Large Language Models (LLMs) has brought about remarkable capabilities in natural language processing but also raised concerns about their potential misuse. While strategies like supervised fine-tuning and reinforcement learning from human feedback have enhanced their safety, these methods primarily focus on natural languages, which may not generalize to other domains. This paper introduces CodeAttack, a framework that transforms natural language inputs into code inputs, presenting a novel environment for testing the safety generalization of LLMs. Our comprehensive studies on state-of-the-art LLMs including GPT-4, Claude-2, and Llama-2 series reveal a common safety vulnerability of these models against code input: CodeAttack consistently bypasses the safety guardrails of all models more than 80% of the time. Furthermore, we find that a larger distribution gap between CodeAttack and natural language leads to weaker safety generalization, such as encoding natural language input with data structures or using less popular programming languages. These findings highlight new safety risks in the code domain and the need for more robust safety alignment algorithms to match the code capabilities of LLMs.

翻译：大语言模型的快速发展带来了自然语言处理的卓越能力，但也引发了对其潜在滥用的担忧。尽管监督微调和基于人类反馈的强化学习等策略增强了模型的安全性，但这些方法主要关注自然语言，可能无法泛化到其他领域。本文提出CodeAttack框架，将自然语言输入转换为代码输入，为测试大语言模型的安全泛化提供了新环境。我们对包括GPT-4、Claude-2和Llama-2系列在内的最先进大语言模型进行的全面研究表明，这些模型在代码输入方面存在普遍的安全漏洞：CodeAttack始终能绕过所有模型的安全防护措施，成功率超过80%。此外，我们发现CodeAttack与自然语言之间的分布差距越大，安全泛化能力越弱，例如使用数据结构编码自然语言输入或采用较少使用的编程语言。这些发现凸显了代码领域的新安全风险，以及需要更鲁棒的安全对齐算法来匹配大语言模型的代码能力。