Annotators are not fungible. Their demographics, life experiences, and backgrounds all contribute to how they label data. However, NLP has only recently considered how annotator identity might influence their decisions. Here, we present POPQUORN (the POtato-Prolific dataset for QUestion-Answering, Offensiveness, text Rewriting, and politeness rating with demographic Nuance). POPQUORN contains 45,000 annotations from 1,484 annotators, drawn from a representative sample regarding sex, age, and race as the US population. Through a series of analyses, we show that annotators' background plays a significant role in their judgments. Further, our work shows that backgrounds not previously considered in NLP (e.g., education), are meaningful and should be considered. Our study suggests that understanding the background of annotators and collecting labels from a demographically balanced pool of crowd workers is important to reduce the bias of datasets. The dataset, annotator background, and annotation interface are available at https://github.com/Jiaxin-Pei/potato-prolific-dataset .

翻译：注释者并非可互换的。他们的人口统计特征、生活经历和背景都会影响其标注数据的方式。然而，自然语言处理领域直到最近才开始关注注释者身份如何影响其决策。本文提出POPQUORN数据集（基于人口统计细微差别的马铃薯-高产问答、攻击性、文本改写与礼貌性评分数据集）。该数据集包含来自1484名注释者的45000条标注，其性别、年龄和种族分布与美国人口具有代表性。通过一系列分析，我们证明注释者的背景对其判断具有显著影响。此外，我们的研究显示，先前在自然语言处理中未被考虑的背景因素（如教育水平）也具有意义且值得关注。本研究建议，理解注释者的背景并从人口统计均衡的众包工作者群体中收集标注数据，对于降低数据集偏差至关重要。数据集、注释者背景信息及标注界面可通过https://github.com/Jiaxin-Pei/potato-prolific-dataset 获取。