In most classification models, it has been assumed to have a single ground truth label for each data point. However, subjective tasks like toxicity classification can lead to genuine disagreement among annotators. In these cases aggregating labels will result in biased labeling and, consequently, biased models that can overlook minority opinions. Previous studies have shed light on the pitfalls of label aggregation and have introduced a handful of practical approaches to tackle this issue. Recently proposed multi-annotator models, which predict labels individually per annotator, are vulnerable to under-determination for annotators with small samples. This problem is especially the case in crowd-sourced datasets. In this work, we propose Annotator Aware Representations for Texts (AART) for subjective classification tasks. We will show the improvement of our method on metrics that assess the performance on capturing annotators' perspectives. Additionally, our approach involves learning representations for annotators, allowing for an exploration of the captured annotation behaviors.

翻译：在大多数分类模型中，通常假设每个数据点存在唯一真实标签。然而，主观性任务（如毒性分类）可能导致标注者之间存在真实分歧。在此类情况下，聚合标签会导致标注偏差，进而产生可能忽略少数意见的偏颇模型。既有研究已揭示标签聚合的缺陷，并提出若干实用解决方法。近期提出的多标注者模型（为每位标注者单独预测标签）对于样本量较小的标注者存在欠定问题——这在众包数据集中尤为突出。本研究提出面向主观分类任务的文本标注者感知表征方法（AART，Annotator Aware Representations for Texts）。我们将展示该方法在评估标注者视角捕捉性能的指标上的改进。此外，本方法通过为标注者学习表征，为探索捕获的标注行为提供了可能。