With adversarial or otherwise normal prompts, existing large language models (LLM) can be pushed to generate toxic discourses. One way to reduce the risk of LLMs generating undesired discourses is to alter the training of the LLM. This can be very restrictive due to demanding computation requirements. Other methods rely on rule-based or prompt-based token elimination, which are limited as they dismiss future tokens and the overall meaning of the complete discourse. Here, we center detoxification on the probability that the finished discourse is ultimately considered toxic. That is, at each point, we advise against token selections proportional to how likely a finished text from this point will be toxic. To this end, we formally extend the dead-end theory from the recent reinforcement learning (RL) literature to also cover uncertain outcomes. Our approach, called rectification, utilizes a separate but significantly smaller model for detoxification, which can be applied to diverse LLMs as long as they share the same vocabulary. Importantly, our method does not require access to the internal representations of the LLM, but only the token probability distribution at each decoding step. This is crucial as many LLMs today are hosted in servers and only accessible through APIs. When applied to various LLMs, including GPT-3, our approach significantly improves the generated discourse compared to the base LLMs and other techniques in terms of both the overall language and detoxification performance.

翻译：面对对抗性或常规提示，现有大型语言模型（LLM）可能生成有害话语。降低LLM生成不当话语风险的一种方式是修改其训练过程，但这因计算需求过高而受到严格限制。其他方法依赖基于规则或提示的令牌消除策略，但由于忽略后续令牌及完整话语的整体语义，其效果有限。本研究将去毒化聚焦于生成话语最终被判定为有害的概率：即在每一步解码中，我们依据从该节点生成完整文本可能有害的概率，反向建议抑制对应令牌的选择。为此，我们将近期强化学习（RL）文献中的"死胡同"理论正式扩展至涵盖不确定结果。我们提出的修正方法利用一个独立但规模显著更小的模型进行去毒化，可适用于任何共享相同词表的LLM。关键的是，本方法无需访问LLM内部表征，仅需每个解码步骤的令牌概率分布——这对当前众多仅通过API访问的云端LLM尤为重要。将本方法应用于包括GPT-3在内的多种LLM后，无论从整体语言质量还是去毒化效果来看，生成话语均显著优于基准LLM及其他已有技术。