We propose a method to control the attributes of Language Models (LMs) for the text generation task using Causal Average Treatment Effect (ATE) scores and counterfactual augmentation. We explore this method, in the context of LM detoxification, and propose the Causally Fair Language (CFL) architecture for detoxifying pre-trained LMs in a plug-and-play manner. Our architecture is based on a Structural Causal Model (SCM) that is mathematically transparent and computationally efficient as compared with many existing detoxification techniques. We also propose several new metrics that aim to better understand the behaviour of LMs in the context of toxic text generation. Further, we achieve state of the art performance for toxic degeneration, which are computed using \RTP (RTP) benchmark. Our experiments show that CFL achieves such a detoxification without much impact on the model perplexity. We also show that CFL mitigates the unintended bias problem through experiments on the BOLD dataset.

翻译：我们提出了一种方法，利用因果平均处理效应（ATE）得分和反事实增强来控制语言模型（LM）在文本生成任务中的属性。我们在语言模型去毒化背景下探索了该方法，并提出了因果公平语言（CFL）架构，以即插即用的方式对预训练语言模型进行去毒化。与许多现有去毒化技术相比，该架构基于结构因果模型（SCM），在数学上具有透明性并计算高效。我们还提出了若干新指标，旨在更深入地理解语言模型在有毒文本生成中的行为。此外，我们使用RTP（RTP）基准测试实现了有毒退化方面的最先进性能。实验表明，CFL能在不显著影响模型困惑度的情况下实现去毒化。同时，通过在BOLD数据集上的实验，我们展示了CFL缓解了非预期偏差问题。