In this paper, we present \textit{JADE}, a targeted linguistic fuzzing platform which strengthens the linguistic complexity of seed questions to simultaneously and consistently break a wide range of widely-used LLMs categorized in three groups: eight open-sourced Chinese, six commercial Chinese and four commercial English LLMs. JADE generates three safety benchmarks for the three groups of LLMs, which contain unsafe questions that are highly threatening: the questions simultaneously trigger harmful generation of multiple LLMs, with an average unsafe generation ratio of \textbf{$70\%$} (please see the table below), while are still natural questions, fluent and preserving the core unsafe semantics. We release the benchmark demos generated for commercial English LLMs and open-sourced English LLMs in the following link: https://github.com/whitzard-ai/jade-db. For readers who are interested in evaluating on more questions generated by JADE, please contact us. \textit{JADE} is based on Noam Chomsky's seminal theory of transformational-generative grammar. Given a seed question with unsafe intention, \textit{JADE} invokes a sequence of generative and transformational rules to increment the complexity of the syntactic structure of the original question, until the safety guardrail is broken. Our key insight is: Due to the complexity of human language, most of the current best LLMs can hardly recognize the invariant evil from the infinite number of different syntactic structures which form an unbound example space that can never be fully covered. Technically, the generative/transformative rules are constructed by native speakers of the languages, and, once developed, can be used to automatically grow and transform the parse tree of a given question, until the guardrail is broken. For more evaluation results and demo, please check our website: https://whitzard-ai.github.io/jade.html.

翻译：本文提出了一种名为**JADE**的针对性语言模糊测试平台，通过增强种子问题的语言复杂度，能够同时且一致地攻破三大类广泛使用的LLM：包括八种开源中文、六种商用中文和四种商用英文LLM。JADE为这三类LLM生成了三个安全基准，其中包含高度威胁性的不安全问题：这些问题能同时触发多个LLM的有害生成，平均不安全生成比例达**70%**（详见下表），同时仍保持自然流畅的问句形式且保留核心不安全语义。我们在以下链接发布了为商用英文LLM和开源英文LLM生成的基准示例：https://github.com/whitzard-ai/jade-db。若读者对使用JADE生成的更多问题开展评估感兴趣，请联系我们。JADE基于诺姆·乔姆斯基的转换生成语法开创性理论。给定一个具有不安全意图的种子问题，JADE会调用一系列生成与转换规则，逐步增加原始问题句法结构的复杂度，直至安全护栏被攻破。我们的核心洞见是：由于人类语言的复杂性，当前大多数最先进的LLM难以从无限的不同句法结构中识别出不变的恶意意图——这些结构构成了一个永远无法完全覆盖的无限示例空间。在技术上，生成/转换规则由母语者构建，一旦开发完成，即可自动扩展和转换给定问题的解析树，直至安全护栏被突破。更多评估结果与示例请访问我们的网站：https://whitzard-ai.github.io/jade.html。