Societal biases are reflected in large pre-trained language models and their fine-tuned versions on downstream tasks. Common in-processing bias mitigation approaches, such as adversarial training and mutual information removal, introduce additional optimization criteria, and update the model to reach a new debiased state. However, in practice, end-users and practitioners might prefer to switch back to the original model, or apply debiasing only on a specific subset of protected attributes. To enable this, we propose a novel modular bias mitigation approach, consisting of stand-alone highly sparse debiasing subnetworks, where each debiasing module can be integrated into the core model on-demand at inference time. Our approach draws from the concept of \emph{diff} pruning, and proposes a novel training regime adaptable to various representation disentanglement optimizations. We conduct experiments on three classification tasks with gender, race, and age as protected attributes. The results show that our modular approach, while maintaining task performance, improves (or at least remains on-par with) the effectiveness of bias mitigation in comparison with baseline finetuning. Particularly on a two-attribute dataset, our approach with separately learned debiasing subnetworks shows effective utilization of either or both the subnetworks for selective bias mitigation.

翻译：摘要：社会偏见反映在大型预训练语言模型及其下游任务的微调版本中。常见的处理中偏差缓解方法，如对抗训练和互信息移除，会引入额外的优化准则，并更新模型以达成新的去偏状态。然而，在实际应用中，最终用户和从业者可能更倾向于切换回原始模型，或仅对特定受保护属性子集应用去偏处理。为实现这一目标，我们提出了一种新颖的模块化偏差缓解方法，该方法由独立的高稀疏性去偏子网络组成，每个去偏模块可在推理时按需集成到核心模型中。我们的方法借鉴了“差异剪枝”的概念，并提出了一种适应多种表示解耦优化的新型训练机制。我们在三个分类任务上进行了实验，将性别、种族和年龄作为受保护属性。结果表明，与基线微调相比，我们的模块化方法在保持任务性能的同时，提升（或至少持平）了偏差缓解的有效性。特别是在一个双属性数据集上，我们采用分别学习的去偏子网络的方法，展示了有效利用任一或两个子网络进行选择性偏差缓解的能力。