Existing studies addressing gender bias of pre-trained language models, usually build a small gender-neutral data set and conduct a second phase pre-training on the model with such data. However, given the limited size and concentrated focus of the gender-neutral data, catastrophic forgetting would occur during second-phase pre-training. Forgetting information in the original training data may damage the model's downstream performance by a large margin. In this work, we empirically show that catastrophic forgetting occurs in such methods by evaluating them with general NLP tasks in GLUE. Then, we propose a new method, GEnder Equality Prompt (GEEP), to improve gender fairness of pre-trained models with less forgetting. GEEP freezes the pre-trained model and learns gender-related prompts with gender-neutral data. Empirical results show that GEEP not only achieves SOTA performances on gender fairness tasks, but also forgets less and performs better on GLUE by a large margin.

翻译：现有研究在解决预训练语言模型的性别偏见问题时，通常构建小规模中性性别数据集，并基于此类数据进行第二阶段预训练。然而，由于中性性别数据的规模有限且内容高度集中，第二阶段预训练过程中会出现灾难性遗忘。原始训练数据中的信息遗忘可能大幅损害模型的下游任务性能。本研究通过GLUE通用自然语言处理任务评估发现，此类方法确实存在灾难性遗忘现象。据此，我们提出新方法——性别平等提示（GEEP），以在减少遗忘的前提下提升预训练模型的性别公平性。GEEP冻结预训练模型，利用中性性别数据学习与性别相关的提示。实验结果表明，GEEP不仅在性别公平任务中达到最优性能，且遗忘程度显著降低，在GLUE任务上的表现也大幅提升。