The increasing size of large language models (LLMs) has introduced challenges in their training and inference. Removing model components is perceived as a solution to tackle the large model sizes, however, existing pruning methods solely focus on performance, without considering an essential aspect for the responsible use of LLMs: model fairness. It is crucial to address the fairness of LLMs towards diverse groups, such as women, Black people, LGBTQ+, Jewish communities, among others, as they are being deployed and available to a wide audience. In this work, first, we investigate how attention heads impact fairness and performance in pre-trained transformer-based language models. We then propose a novel method to prune the attention heads that negatively impact fairness while retaining the heads critical for performance, i.e. language modeling capabilities. Our approach is practical in terms of time and resources, as it does not require fine-tuning the final pruned, and fairer, model. Our findings demonstrate a reduction in gender bias by 19%, 19.5%, 39.5%, 34.7%, 23%, and 8% for DistilGPT-2, GPT-2, GPT-Neo of two different sizes, GPT-J, and Llama 2 models, respectively, in comparison to the biased model, with only a slight decrease in performance.

翻译：大语言模型（LLMs）规模的持续增长为其训练与推理带来了挑战。移除模型组件被视为应对大模型规模的解决方案，然而现有剪枝方法仅关注性能，未考虑负责任使用LLMs的关键要素：模型公平性。随着LLMs被部署并向广泛受众开放，亟需解决其对不同群体（如女性、黑人、LGBTQ+群体、犹太社群等）的公平性问题。本研究首先探究了注意力头对预训练Transformer语言模型公平性与性能的影响，继而提出一种新型剪枝方法——在保留对语言建模能力至关重要的注意力头的同时，剪除对公平性产生负面影响的头。该方法无需对最终剪枝后的更公平模型进行微调，在时间与资源层面具有实用性。实验表明，相较于存在偏见的基线模型，DistilGPT-2、GPT-2、两种不同规模的GPT-Neo、GPT-J及Llama 2模型的性别偏见分别降低了19%、19.5%、39.5%、34.7%、23%和8%，而性能仅略有下降。