Recent advancements in generative AI have enabled ubiquitous access to large language models (LLMs). Empowered by their exceptional capabilities to understand and generate human-like text, these models are being increasingly integrated into our society. At the same time, there are also concerns on the potential misuse of this powerful technology, prompting defensive measures from service providers. To overcome such protection, jailbreaking prompts have recently emerged as one of the most effective mechanisms to circumvent security restrictions and elicit harmful content originally designed to be prohibited. Due to the rapid development of LLMs and their ease of access via natural languages, the frontline of jailbreak prompts is largely seen in online forums and among hobbyists. To gain a better understanding of the threat landscape of semantically meaningful jailbreak prompts, we systemized existing prompts and measured their jailbreak effectiveness empirically. Further, we conducted a user study involving 92 participants with diverse backgrounds to unveil the process of manually creating jailbreak prompts. We observed that users often succeeded in jailbreak prompts generation regardless of their expertise in LLMs. Building on the insights from the user study, we also developed a system using AI as the assistant to automate the process of jailbreak prompt generation.

翻译：近期生成式人工智能的进展使得大规模语言模型得以普及。凭借其理解和生成类人文本的卓越能力，这些模型正日益融入我们的社会。然而，这一强大技术也可能被滥用，这促使服务提供商采取防御措施。为突破此类保护机制，越狱提示近年来已成为规避安全限制、生成原本被禁止的有害内容的最有效手段之一。由于大语言模型发展迅速且可通过自然语言轻松访问，越狱提示的前沿阵地主要出现在在线论坛和爱好者群体中。为深入理解语义性越狱提示的威胁态势，本文对现有提示进行了系统化分类，并通过实证评估了其越狱效果。此外，我们开展了一项涉及92名不同背景参与者的用户研究，揭示了手动创建越狱提示的过程。研究发现，无论用户对LLM的专业知识如何，他们通常都能成功生成越狱提示。基于用户研究的洞察，我们进一步开发了一种以AI为辅助的系统，用于自动化生成越狱提示。