The increasing use of foundation models highlights the urgent need to address and eliminate implicit biases present in them that arise during pretraining. In this paper, we introduce PEFTDebias, a novel approach that employs parameter-efficient fine-tuning (PEFT) to mitigate the biases within foundation models. PEFTDebias consists of two main phases: an upstream phase for acquiring debiasing parameters along a specific bias axis, and a downstream phase where these parameters are incorporated into the model and frozen during the fine-tuning process. By evaluating on four datasets across two bias axes namely gender and race, we find that downstream biases can be effectively reduced with PEFTs. In addition, we show that these parameters possess axis-specific debiasing characteristics, enabling their effective transferability in mitigating biases in various downstream tasks. To ensure reproducibility, we release the code to do our experiments.

翻译：基础模型日益广泛的使用凸显了解决并消除其在预训练过程中产生的隐性偏见的迫切需求。本文提出了PEFTDebias，一种利用参数高效微调（PEFT）来缓解基础模型偏见的新方法。PEFTDebias包含两个主要阶段：上游阶段，用于沿特定偏见轴获取去偏参数；下游阶段，将这些参数融入模型并在微调过程中冻结。通过在涵盖性别和种族两个偏见轴的四个数据集上进行评估，我们发现PEFT能有效降低下游偏见。此外，我们证明这些参数具有轴特定的去偏特性，使其能够在多种下游任务中有效迁移以缓解偏见。为确保可重复性，我们公开了实验代码。