Societal biases present in pre-trained large language models are a critical issue as these models have been shown to propagate biases in countless downstream applications, rendering them unfair towards specific groups of people. Since large-scale retraining of these models from scratch is both time and compute-expensive, a variety of approaches have been previously proposed that de-bias a pre-trained model. While the majority of current state-of-the-art debiasing methods focus on changes to the training regime, in this paper, we propose data intervention strategies as a powerful yet simple technique to reduce gender bias in pre-trained models. Specifically, we empirically show that by fine-tuning a pre-trained model on only 10 de-biased (intervened) training examples, the tendency to favor any gender is significantly reduced. Since our proposed method only needs a few training examples, our few-shot debiasing approach is highly feasible and practical. Through extensive experimentation, we show that our debiasing technique performs better than competitive state-of-the-art baselines with minimal loss in language modeling ability.

翻译：预训练大语言模型中存在的社会偏见是关键问题，因为这些模型已被证明会在无数下游应用中传播偏见，导致对特定群体产生不公平对待。由于从头开始大规模重新训练这些模型既耗时又昂贵，此前已提出多种方法对预训练模型进行去偏。尽管当前最先进的去偏方法主要关注训练机制的改变，但本文提出数据干预策略作为减少预训练模型性别偏见的强大而简单的技术。具体而言，我们通过实验证明，仅对10个去偏（干预）训练样本微调预训练模型，就能显著降低对任何性别的偏好倾向。由于所提方法仅需少量训练样本，这种小样本去偏方法具有高度可行性和实用性。通过广泛实验表明，我们的去偏技术在语言建模能力损失极小的情况下，性能优于竞争性最先进基线方法。