Demographics and cultural background of annotators influence the labels they assign in text annotation -- for instance, an elderly woman might find it offensive to read a message addressed to a "bro", but a male teenager might find it appropriate. It is therefore important to acknowledge label variations to not under-represent members of a society. Two research directions developed out of this observation in the context of using large language models (LLM) for data annotations, namely (1) studying biases and inherent knowledge of LLMs and (2) injecting diversity in the output by manipulating the prompt with demographic information. We combine these two strands of research and ask the question to which demographics an LLM resorts to when no demographics is given. To answer this question, we evaluate which attributes of human annotators LLMs inherently mimic. Furthermore, we compare non-demographic conditioned prompts and placebo-conditioned prompts (e.g., "you are an annotator who lives in house number 5") to demographics-conditioned prompts ("You are a 45 year old man and an expert on politeness annotation. How do you rate {instance}"). We study these questions for politeness and offensiveness annotations on the POPQUORN data set, a corpus created in a controlled manner to investigate human label variations based on demographics which has not been used for LLM-based analyses so far. We observe notable influences related to gender, race, and age in demographic prompting, which contrasts with previous studies that found no such effects.

翻译：标注者的人口统计学特征与文化背景会影响其在文本标注中分配的标签——例如，一位年长女性可能认为称呼"兄弟"的信息具有冒犯性，而男性青少年则可能认为其恰当。因此，承认标签的多样性对于避免社会成员被忽视至关重要。基于这一观察，在使用大语言模型进行数据标注的研究中发展出两个方向：（1）研究大语言模型的偏见与内在知识；（2）通过提示词注入人口统计学信息以增加输出多样性。本研究整合这两个研究方向，探讨当未提供人口统计学信息时，大语言模型会默认采用何种特征。为解答此问题，我们评估了大语言模型内在模仿人类标注者的哪些属性。此外，我们比较了无人口统计学条件提示、安慰剂条件提示（如"你是住在5号房屋的标注者"）与人口统计学条件提示（如"你是一名45岁男性，是礼貌标注专家。请评价{实例}"）。我们基于POPQUORN数据集对礼貌性与冒犯性标注展开研究，该语料库以受控方式构建，旨在探究基于人口统计学的人类标签差异，此前尚未被用于大语言模型分析。研究发现，在人口统计学提示条件下，性别、种族与年龄均产生显著影响，这与先前未发现此类效应的研究形成鲜明对比。