Large language models are becoming the go-to solution for the ever-growing number of tasks. However, with growing capacity, models are prone to rely on spurious correlations stemming from biases and stereotypes present in the training data. This work proposes a novel method for detecting and mitigating gender bias in language models. We perform causal analysis to identify problematic model components and discover that mid-upper feed-forward layers are most prone to convey bias. Based on the analysis results, we intervene in the model by applying a linear projection to the weight matrices of these layers. Our titular method, DAMA, significantly decreases bias as measured by diverse metrics while maintaining the model's performance on downstream tasks. We release code for our method and models, which retrain LLaMA's state-of-the-art performance while being significantly less biased.

翻译：大型语言模型正日益成为处理不断增长任务数量的首选解决方案。然而，随着模型容量的增加，它们容易依赖训练数据中存在的偏见和刻板印象所产生的伪相关性。本研究提出了一种检测和缓解语言模型中性别偏见的新方法。我们通过因果分析识别有问题的模型组件，发现中上层前馈层最容易传递偏见。基于分析结果，我们通过对这些层的权重矩阵施加线性投影来干预模型。我们提出的DAMA方法在多种度量标准下显著降低了偏见，同时保持了模型在下游任务上的性能。我们发布了方法和模型的代码，这些模型在保持LLaMA先进性能的同时显著减少了偏见。