We investigate in this paper how distributions of occupations with respect to gender is reflected in pre-trained language models. Such distributions are not always aligned to normative ideals, nor do they necessarily reflect a descriptive assessment of reality. In this paper, we introduce an approach for measuring to what degree pre-trained language models are aligned to normative and descriptive occupational distributions. To this end, we use official demographic information about gender--occupation distributions provided by the national statistics agencies of France, Norway, United Kingdom, and the United States. We manually generate template-based sentences combining gendered pronouns and nouns with occupations, and subsequently probe a selection of ten language models covering the English, French, and Norwegian languages. The scoring system we introduce in this work is language independent, and can be used on any combination of template-based sentences, occupations, and languages. The approach could also be extended to other dimensions of national census data and other demographic variables.

翻译：本文研究预训练语言模型中职业按性别的分布情况。这类分布既不一定符合规范性理想，也不一定反映对现实的描述性评估。我们提出了一种衡量预训练语言模型与规范性和描述性职业分布对齐程度的方法。为此，我们使用法国、挪威、英国和美国国家统计机构提供的官方人口统计信息（性别-职业分布）。我们手动生成基于模板的句子，将性别代词和名词与职业相结合，随后探查十种覆盖英语、法语和挪威语的语言模型。本文引入的评分系统与语言无关，可适用于任何模板句子、职业和语言的组合。该方法还可扩展到国家人口普查数据的其他维度及其他人口统计变量。