Large Language Models (LLMs) trained with self-supervision on vast corpora of web text fit to the social biases of that text. Without intervention, these social biases persist in the model's predictions in downstream tasks, leading to representational harm. Many strategies have been proposed to mitigate the effects of inappropriate social biases learned during pretraining. Simultaneously, methods for model compression have become increasingly popular to reduce the computational burden of LLMs. Despite the popularity and need for both approaches, little work has been done to explore the interplay between these two. We perform a carefully controlled study of the impact of model compression via quantization and knowledge distillation on measures of social bias in LLMs. Longer pretraining and larger models led to higher social bias, and quantization showed a regularizer effect with its best trade-off around 20% of the original pretraining time.

翻译：大型语言模型通过自监督学习在海量网络文本语料上训练，会拟合这些文本中的社会偏见。若不加以干预，这些社会偏见将持续存在于模型在下游任务中的预测结果中，造成表征性伤害。已有多种策略被提出用于缓解预训练过程中习得的不当社会偏见影响。与此同时，模型压缩方法因能减轻大型语言模型的计算负担而日益流行。尽管这两类方法均具有广泛需求且备受关注，但关于二者交互作用的研究仍十分有限。我们通过精心控制的实验，研究了通过量化和知识蒸馏实现的模型压缩对大型语言模型社会偏见测量指标的影响。更长的预训练时间和更大的模型规模会导致更高的社会偏见，而量化展现出正则化效应，其最佳平衡点出现在原始预训练时间的约20%处。