Stereotype benchmark datasets are crucial to detect and mitigate social stereotypes about groups of people in NLP models. However, existing datasets are limited in size and coverage, and are largely restricted to stereotypes prevalent in the Western society. This is especially problematic as language technologies gain hold across the globe. To address this gap, we present SeeGULL, a broad-coverage stereotype dataset, built by utilizing generative capabilities of large language models such as PaLM, and GPT-3, and leveraging a globally diverse rater pool to validate the prevalence of those stereotypes in society. SeeGULL is in English, and contains stereotypes about identity groups spanning 178 countries across 8 different geo-political regions across 6 continents, as well as state-level identities within the US and India. We also include fine-grained offensiveness scores for different stereotypes and demonstrate their global disparities. Furthermore, we include comparative annotations about the same groups by annotators living in the region vs. those that are based in North America, and demonstrate that within-region stereotypes about groups differ from those prevalent in North America. CONTENT WARNING: This paper contains stereotype examples that may be offensive.

翻译：刻板印象基准数据集对于检测和减轻自然语言处理模型中关于人群的社会刻板印象至关重要。然而，现有数据集在规模和覆盖范围上有限，且主要局限于西方社会中流行的刻板印象。随着语言技术在全球范围内获得应用，这一问题尤为突出。为填补这一空白，我们提出了SeeGULL——一个广泛覆盖的刻板印象数据集。该数据集利用PaLM、GPT-3等大型语言模型的生成能力，并借助全球多样化的评估者池来验证这些刻板印象在社会中的普遍性。SeeGULL数据集为英文，涵盖了分布于六大洲八个不同地缘政治区域的178个国家的身份群体，以及美国和印度国内各州层面的身份群体。我们还提供了不同刻板印象的细粒度冒犯性评分，并揭示了其全球差异。此外，我们比较了来自区域内部与北美地区评估者关于相同群体的标注结果，表明区域内关于群体的刻板印象与北美地区流行的刻板印象存在差异。内容警告：本文包含可能令人不适的刻板印象示例。