The open-endedness of large language models (LLMs) combined with their impressive capabilities may lead to new safety issues when being exploited for malicious use. While recent studies primarily focus on probing toxic outputs that can be easily detected with existing toxicity classifiers, we show that LLMs can generate diverse implicit toxic outputs that are exceptionally difficult to detect via simply zero-shot prompting. Moreover, we propose a reinforcement learning (RL) based attacking method to further induce the implicit toxicity in LLMs. Specifically, we optimize the language model with a reward that prefers implicit toxic outputs to explicit toxic and non-toxic ones. Experiments on five widely-adopted toxicity classifiers demonstrate that the attack success rate can be significantly improved through RL fine-tuning. For instance, the RL-finetuned LLaMA-13B model achieves an attack success rate of 90.04% on BAD and 62.85% on Davinci003. Our findings suggest that LLMs pose a significant threat in generating undetectable implicit toxic outputs. We further show that fine-tuning toxicity classifiers on the annotated examples from our attacking method can effectively enhance their ability to detect LLM-generated implicit toxic language. The code is publicly available at https://github.com/thu-coai/Implicit-Toxicity.

翻译：大型语言模型的开放性及其令人印象深刻的强大能力，在遭受恶意利用时可能引发新的安全问题。尽管近期研究主要聚焦于探测那些可通过现有毒性分类器轻易识别的显性毒性输出，但我们发现，通过简单的零样本提示，大型语言模型能够生成各种难以检测的隐性毒性输出。此外，我们提出了一种基于强化学习的攻击方法，以进一步诱导大型语言模型中的隐性毒性。具体而言，我们利用一种偏好隐性毒性输出而非显性毒性及非毒性输出的奖励机制对语言模型进行优化。在五个广泛采用的毒性分类器上进行的实验表明，通过强化学习微调，攻击成功率可显著提升。例如，经过强化学习微调的LLaMA-13B模型在BAD和Davinci003上分别达到了90.04%和62.85%的攻击成功率。我们的研究结果表明，大型语言模型在生成难以检测的隐性毒性输出方面构成了重大威胁。我们进一步证明，利用攻击方法中的标注示例对毒性分类器进行微调，可有效提升其检测大型语言模型生成的隐性毒性语言的能力。相关代码已在https://github.com/thu-coai/Implicit-Toxicity公开。