Mitigation of gender bias in NLP has a long history tied to debiasing static word embeddings. More recently, attention has shifted to debiasing pre-trained language models. We study to what extent the simplest projective debiasing methods, developed for word embeddings, can help when applied to BERT's internal representations. Projective methods are fast to implement, use a small number of saved parameters, and make no updates to the existing model parameters. We evaluate the efficacy of the methods in reducing both intrinsic bias, as measured by BERT's next sentence prediction task, and in mitigating observed bias in a downstream setting when fine-tuned. To this end, we also provide a critical analysis of a popular gender-bias assessment test for quantifying intrinsic bias, resulting in an enhanced test set and new bias measures. We find that projective methods can be effective at both intrinsic bias and downstream bias mitigation, but that the two outcomes are not necessarily correlated. This finding serves as a warning that intrinsic bias test sets, based either on language modeling tasks or next sentence prediction, should not be the only benchmark in developing a debiased language model.

翻译：自然语言处理中性别偏见的缓解有着悠久的历史，与静态词嵌入的去偏方法紧密相关。近年来，注意力已转向预训练语言模型的去偏。我们研究了为词嵌入开发的最简单的投影去偏方法在应用于BERT内部表示时能在多大程度上发挥作用。投影方法实现快速，使用少量保存的参数，且不对现有模型参数进行更新。我们评估了这些方法在减少内在偏见（通过BERT的下一句预测任务衡量）以及在下游微调场景中缓解观察到的偏见方面的效果。为此，我们对一种流行的用于量化内在偏见的性别偏见评估测试进行了批判性分析，从而形成了一个增强的测试集和新的偏见度量。我们发现投影方法在内在偏见和下游偏见的缓解方面都能有效，但这两个结果不一定相关。这一发现警示我们，基于语言建模任务或下一句预测的内在偏见测试集不应成为开发去偏语言模型时唯一的标准。