Despite extensive safety measures, LLMs are vulnerable to adversarial inputs, or jailbreaks, which can elicit unsafe behaviors. In this work, we introduce bijection learning, a powerful attack algorithm which automatically fuzzes LLMs for safety vulnerabilities using randomly-generated encodings whose complexity can be tightly controlled. We leverage in-context learning to teach models bijective encodings, pass encoded queries to the model to bypass built-in safety mechanisms, and finally decode responses back into English. Our attack is extremely effective on a wide range of frontier language models. Moreover, by controlling complexity parameters such as number of key-value mappings in the encodings, we find a close relationship between the capability level of the attacked LLM and the average complexity of the most effective bijection attacks. Our work highlights that new vulnerabilities in frontier models can emerge with scale: more capable models are more severely jailbroken by bijection attacks.

翻译：尽管已采取广泛的安全措施，大型语言模型仍易受对抗性输入（即越狱攻击）的影响，从而引发不安全行为。本研究提出双射学习——一种强大的攻击算法，该算法利用随机生成的编码自动对大型语言模型进行安全漏洞模糊测试，且编码复杂度可被严格调控。我们利用上下文学习使模型掌握双射编码，将编码后的查询输入模型以绕过内置安全机制，最终将响应解码回英文。该攻击对各类前沿语言模型均表现出极高有效性。此外，通过控制编码中键值映射数量等复杂度参数，我们发现被攻击大型语言模型的能力水平与最有效双射攻击的平均复杂度之间存在密切关联。本研究揭示前沿模型的新漏洞可能随规模扩大而显现：能力越强的模型受双射攻击的越狱程度越严重。