In recent years, the rapid development of large language models (LLMs) has brought new vitality into various domains, generating substantial social and economic benefits. However, this swift advancement has also introduced new security vulnerabilities. Jailbreaking, a form of attack that induces LLMs to produce harmful content through carefully crafted prompts, presents a significant challenge to the safe and trustworthy development of LLMs. Previous jailbreak methods primarily exploited the internal properties or capabilities of LLMs, such as optimization-based jailbreak approaches and methods that leveraged the model's context-learning abilities. In this paper, we introduce a novel jailbreak method, SQL Injection Jailbreak (SIJ), which targets the external properties of LLMs, specifically, the way LLMs construct input prompts. By injecting jailbreak information into user prompts, SIJ successfully induces the model to output harmful content. Our SIJ method achieves near 100\% attack success rates on five well-known open-source LLMs on the AdvBench, while incurring lower time costs compared to previous methods. More importantly, SIJ is the first method to exploit the external properties of LLMs for jailbreak attacks and exposes a new vulnerability in LLMs that urgently requires mitigation. To address this, we propose a simple defense method called Self-Reminder-Key to counter SIJ and demonstrate its effectiveness through experimental results. Our code is available at \href{https://github.com/weiyezhimeng/SQL-Injection-Jailbreak}{https://github.com/weiyezhimeng/SQL-Injection-Jailbreak}.

翻译：近年来，大语言模型（LLMs）的快速发展为各领域注入了新的活力，产生了巨大的社会经济效益。然而，这种快速进步也带来了新的安全漏洞。越狱攻击作为一种通过精心设计的提示诱导LLMs生成有害内容的攻击形式，对LLMs安全可信的发展构成了重大挑战。以往的越狱方法主要利用LLMs的内部特性或能力，例如基于优化的越狱方法以及利用模型上下文学习能力的方法。本文提出一种新型越狱方法——SQL注入越狱（SIJ），该方法针对LLMs的外部特性，特别是LLMs构建输入提示的方式。通过向用户提示中注入越狱信息，SIJ成功诱导模型输出有害内容。我们的SIJ方法在AdvBench基准上对五个知名开源LLMs实现了接近100%的攻击成功率，同时相比先前方法具有更低的时间成本。更重要的是，SIJ是首个利用LLMs外部特性进行越狱攻击的方法，揭示了LLMs中亟待修补的新漏洞。为此，我们提出一种名为“自我提醒密钥”的简单防御方法来应对SIJ，并通过实验结果验证了其有效性。我们的代码发布于 \href{https://github.com/weiyezhimeng/SQL-Injection-Jailbreak}{https://github.com/weiyezhimeng/SQL-Injection-Jailbreak}。