Detoxification for LLMs is challenging since it requires models to avoid generating harmful content while maintaining the generation capability. To ensure the safety of generations, previous detoxification methods detoxify the models by changing the data distributions or constraining the generations from different aspects in a single-step manner. However, these approaches will dramatically affect the generation quality of LLMs, e.g., discourse coherence and semantic consistency, since language models tend to generate along the toxic prompt while detoxification methods work in the opposite direction. To handle such a conflict, we decompose the detoxification process into different sub-steps, where the detoxification is concentrated in the input stage and the subsequent continual generation is based on the non-toxic prompt. Besides, we also calibrate the strong reasoning ability of LLMs by designing a Detox-Chain to connect the above sub-steps in an orderly manner, which allows LLMs to detoxify the text step-by-step. Automatic and human evaluation on two benchmarks reveals that by training with Detox-Chain, six LLMs scaling from 1B to 33B can obtain significant detoxification and generation improvement. Our code and data are available at https://github.com/CODINNLG/Detox-CoT. Warning: examples in the paper may contain uncensored offensive content.

翻译：大型语言模型的去毒化具有挑战性，因为需要在避免生成有害内容的同时保持生成能力。为确保生成内容的安全性，以往的去毒化方法通过改变数据分布或从不同方面以单步方式约束生成内容来对模型进行去毒。然而，这些方法会显著影响语言模型的生成质量（例如语篇连贯性和语义一致性），因为语言模型倾向于沿毒性提示生成内容，而去毒化方法的作用方向与之相反。为解决这一冲突，我们将去毒过程分解为不同子步骤：去毒集中在输入阶段，后续的连续生成基于无毒提示。此外，我们通过设计去毒链（Detox-Chain）有序连接上述子步骤，校准语言模型强大的推理能力，使其能够逐步对文本进行去毒。在两个基准上的自动和人工评估表明，通过使用去毒链训练，六个规模从1B到33B的语言模型在去毒和生成能力方面均获得显著提升。我们的代码和数据已开源至https://github.com/CODINNLG/Detox-CoT。警告：论文中的示例可能包含未经审查的攻击性内容。