Annotators are not fungible. Their demographics, life experiences, and backgrounds all contribute to how they label data. However, NLP has only recently considered how annotator identity might influence their decisions. Here, we present POPQUORN (the POtato-Prolific dataset for QUestion-Answering, Offensiveness, text Rewriting, and politeness rating with demographic Nuance). POPQUORN contains 45,000 annotations from 1,484 annotators, drawn from a representative sample regarding sex, age, and race as the US population. Through a series of analyses, we show that annotators' background plays a significant role in their judgments. Further, our work shows that backgrounds not previously considered in NLP (e.g., education), are meaningful and should be considered. Our study suggests that understanding the background of annotators and collecting labels from a demographically balanced pool of crowd workers is important to reduce the bias of datasets. The dataset, annotator background, and annotation interface are available at https://github.com/Jiaxin-Pei/potato-prolific-dataset .

翻译：摘要：注释者是不可互换的。他们的年龄、生活经历和背景都会影响他们标注数据的方式。然而，自然语言处理（NLP）领域直到最近才开始考虑注释者身份对其决策的影响。本文提出POPQUORN数据集（面向问答、冒犯性、文本改写和礼貌评级的土豆-众包数据集，包含人口统计学细微差异）。该数据集包含来自1484名注释者的45000条标注，这些注释者按照性别、年龄和种族与美国人口构成具有代表性。通过一系列分析，我们证明注释者的背景在其判断中发挥着重要作用。此外，我们的研究表明，此前NLP中未考虑的背景因素（如教育背景）具有实际意义，应当纳入考量。本研究提示，理解注释者背景并从人口统计学平衡的众包工人池中收集标签，对于减少数据集偏差至关重要。数据集、注释者背景及标注界面可通过https://github.com/Jiaxin-Pei/potato-prolific-dataset获取。