In a world increasingly reliant on artificial intelligence, it is more important than ever to consider the ethical implications of artificial intelligence on humanity. One key under-explored challenge is labeler bias, which can create inherently biased datasets for training and subsequently lead to inaccurate or unfair decisions in healthcare, employment, education, and law enforcement. Hence, we conducted a study to investigate and measure the existence of labeler bias using images of people from different ethnicities and sexes in a labeling task. Our results show that participants possess stereotypes that influence their decision-making process and that labeler demographics impact assigned labels. We also discuss how labeler bias influences datasets and, subsequently, the models trained on them. Overall, a high degree of transparency must be maintained throughout the entire artificial intelligence training process to identify and correct biases in the data as early as possible.

翻译：在一个日益依赖人工智能的世界里，考虑人工智能对人类伦理的影响比以往任何时候都更为重要。其中一个尚未充分探索的关键挑战是标注者偏差，它可能产生本质上存在偏差的训练数据集，进而导致医疗保健、就业、教育和执法等领域出现不准确或不公平的决策。为此，我们开展了一项研究，通过在标注任务中使用来自不同种族和性别的人像图片，来探究并衡量标注者偏差的存在。我们的结果表明，参与者拥有影响其决策过程的刻板印象，并且标注者的人口统计学特征会影响所分配的标签。我们还讨论了标注者偏差如何影响数据集以及随后基于这些数据集训练的模型。总体而言，必须在整个人工智能训练过程中保持高度透明，以便尽早识别和纠正数据中的偏差。