Language models are trained on large-scale corpora that embed implicit biases documented in psychology. Valence associations (pleasantness/unpleasantness) of social groups determine the biased attitudes towards groups and concepts in social cognition. Building on this established literature, we quantify how social groups are valenced in English language models using a sentence template that provides an intersectional context. We study biases related to age, education, gender, height, intelligence, literacy, race, religion, sex, sexual orientation, social class, and weight. We present a concept projection approach to capture the valence subspace through contextualized word embeddings of language models. Adapting the projection-based approach to embedding association tests that quantify bias, we find that language models exhibit the most biased attitudes against gender identity, social class, and sexual orientation signals in language. We find that the largest and better-performing model that we study is also more biased as it effectively captures bias embedded in sociocultural data. We validate the bias evaluation method by overperforming on an intrinsic valence evaluation task. The approach enables us to measure complex intersectional biases as they are known to manifest in the outputs and applications of language models that perpetuate historical biases. Moreover, our approach contributes to design justice as it studies the associations of groups underrepresented in language such as transgender and homosexual individuals.

翻译：语言模型在嵌入心理学中已证明的隐性偏见的大规模语料库上进行训练。社会群体的效价关联（愉悦/不愉悦）决定了社会认知中对群体和概念的偏见态度。基于这一已有文献，我们使用提供交叉语境的句子模板，量化英语语言模型中社会群体的效价。我们研究与年龄、教育、性别、身高、智力、读写能力、种族、宗教、性别、性取向、社会阶层和体重相关的偏见。我们提出一种概念投影方法，通过语言模型的上下文词嵌入捕捉效价子空间。将基于投影的方法适配于量化偏见的嵌入关联测试，我们发现语言模型对语言中的性别认同、社会阶层和性取向信号表现出最具偏见的负面态度。我们发现，我们研究的最大的性能最优模型也更具偏见，因为它有效捕捉了社会文化数据中嵌入的偏见。我们通过在一个内在效价评估任务上取得更优表现来验证该偏见评估方法。该方法使我们能够测量复杂的交叉偏见，这些偏见已知会体现在语言模型的输出和应用中，从而延续历史偏见。此外，我们的方法有助于设计正义，因为它研究了语言中代表性不足群体（如跨性别者和同性恋者）的关联。