Understanding different human attributes and how they affect model behavior may become a standard need for all model creation and usage, from traditional computer vision tasks to the newest multimodal generative AI systems. In computer vision specifically, we have relied on datasets augmented with perceived attribute signals (e.g., gender presentation, skin tone, and age) and benchmarks enabled by these datasets. Typically labels for these tasks come from human annotators. However, annotating attribute signals, especially skin tone, is a difficult and subjective task. Perceived skin tone is affected by technical factors, like lighting conditions, and social factors that shape an annotator's lived experience. This paper examines the subjectivity of skin tone annotation through a series of annotation experiments using the Monk Skin Tone (MST) scale, a small pool of professional photographers, and a much larger pool of trained crowdsourced annotators. Along with this study we release the Monk Skin Tone Examples (MST-E) dataset, containing 1515 images and 31 videos spread across the full MST scale. MST-E is designed to help train human annotators to annotate MST effectively. Our study shows that annotators can reliably annotate skin tone in a way that aligns with an expert in the MST scale, even under challenging environmental conditions. We also find evidence that annotators from different geographic regions rely on different mental models of MST categories resulting in annotations that systematically vary across regions. Given this, we advise practitioners to use a diverse set of annotators and a higher replication count for each image when annotating skin tone for fairness research.

翻译：理解不同人类属性及其对模型行为的影响，可能成为所有模型创建与使用的标准需求——从传统计算机视觉任务到最新多模态生成式AI系统。具体到计算机视觉领域，我们长期依赖通过感知属性信号（如性别呈现、肤色和年龄）增强的数据集，以及基于这些数据集构建的基准测试。这些任务的标注通常由人类标注员完成。然而，标注属性信号（尤其是肤色）是一项困难且具有主观性的任务。感知肤色既受光照条件等技术因素影响，也受标注员生活经历所塑造的社会因素影响。本文通过一系列标注实验探究肤色的主观性，实验中采用Monk肤色量表（MST）、少量专业摄影师及大量经过训练的众包标注员。配合此项研究，我们发布了Monk肤色示例（MST-E）数据集，包含覆盖MST全量表的1515张图像和31段视频。MST-E旨在帮助训练人类标注员有效运用MST量表进行标注。研究表明，即使面对具有挑战性的环境条件，标注员仍能可靠地标注肤色，且结果与MST量表专家高度一致。我们还发现，不同地理区域的标注员对MST类别的心理模型存在差异，导致标注结果呈现系统性区域差异。基于此，我们建议从业者在开展公平性研究的肤色标注时，应采用多元化标注团队，并为每张图像设置更高的重复标注次数。