While Large Language Models (LLMs) display versatile functionality, they continue to generate harmful, biased, and toxic content, as demonstrated by the prevalence of human-designed jailbreaks. In this work, we present Tree of Attacks with Pruning (TAP), an automated method for generating jailbreaks that only requires black-box access to the target LLM. TAP utilizes an LLM to iteratively refine candidate (attack) prompts using tree-of-thoughts reasoning until one of the generated prompts jailbreaks the target. Crucially, before sending prompts to the target, TAP assesses them and prunes the ones unlikely to result in jailbreaks. Using tree-of-thought reasoning allows TAP to navigate a large search space of prompts and pruning reduces the total number of queries sent to the target. In empirical evaluations, we observe that TAP generates prompts that jailbreak state-of-the-art LLMs (including GPT4 and GPT4-Turbo) for more than 80% of the prompts using only a small number of queries. This significantly improves upon the previous state-of-the-art black-box method for generating jailbreaks.

翻译：尽管大语言模型（LLMs）展现了广泛的功能，但它们仍会生成有害、偏见和有毒的内容，人为设计的越狱攻击的普遍性便证明了这一点。本文提出带有剪枝的“攻击之树”（Tree of Attacks with Pruning, TAP），一种自动生成越狱攻击的方法，仅需对目标LLM的黑盒访问权限。TAP利用LLM，通过思维树推理（tree-of-thoughts reasoning）迭代优化候选攻击提示（prompt），直至某个生成的提示成功越狱目标。关键之处在于，在向目标发送提示前，TAP会对其进行评估，并剪除那些不太可能成功越狱的提示。使用思维树推理使TAP能够遍历巨大的提示搜索空间，而剪枝则减少了发送给目标的总查询次数。在实证评估中，我们观察到TAP生成的提示能够以少量查询次数越狱超过80%的当前最先进LLM（包括GPT4和GPT4-Turbo）。这显著改进了此前生成越狱攻击的黑盒方法。