Large language models are becoming the go-to solution for the ever-growing number of tasks. However, with growing capacity, models are prone to rely on spurious correlations stemming from biases and stereotypes present in the training data. This work proposes a novel method for detecting and mitigating gender bias in language models. We perform causal analysis to identify problematic model components and discover that mid-upper feed-forward layers are most prone to convey bias. Based on the analysis results, we intervene in the model by applying a linear projection to the weight matrices of these layers. Our titular method, DAMA, significantly decreases bias as measured by diverse metrics while maintaining the model's performance on downstream tasks. We release code for our method and models, which retrain LLaMA's state-of-the-art performance while being significantly less biased.

翻译：大语言模型正成为处理日益增长任务的优选方案。然而，随着模型能力的提升，模型容易依赖训练数据中存在的偏见和刻板印象所导致的虚假关联。本文提出了一种检测和缓解语言模型中性别偏见的新方法。我们通过因果分析识别有问题的模型组件，发现中上层前馈层最易传递偏见。基于分析结果，我们通过向这些层的权重矩阵施加线性投影来进行模型干预。我们的同名方法DAMA在保持模型下游任务性能的同时，显著降低了多种指标衡量的偏见水平。我们公开了方法和模型的代码，这些方法在保持LLaMA最先进性能的同时显著减少了偏见。