Subjective NLP tasks usually rely on human annotations provided by multiple annotators, whose judgments may vary due to their diverse backgrounds and life experiences. Traditional methods often aggregate multiple annotations into a single ground truth, disregarding the diversity in perspectives that arises from annotator disagreement. In this preliminary study, we examine the effect of including multiple annotations on model accuracy in classification. Our methodology investigates the performance of perspective-aware classification models in stance detection task and further inspects if annotator disagreement affects the model confidence. The results show that multi-perspective approach yields better classification performance outperforming the baseline which uses the single label. This entails that designing more inclusive perspective-aware AI models is not only an essential first step in implementing responsible and ethical AI, but it can also achieve superior results than using the traditional approaches.

翻译：主观性自然语言处理任务通常依赖于由多位标注者提供的人工标注，这些标注者的判断可能因其不同的背景和生活经历而存在差异。传统方法通常将多个标注聚合为单一真实标签，忽视了因标注者分歧而产生的视角多样性。在本初步研究中，我们考察了在分类任务中包含多标注对模型准确性的影响。我们的方法研究了视角感知分类模型在立场检测任务中的表现，并进一步检验了标注者分歧是否影响模型置信度。结果表明，多视角方法能产生更好的分类性能，优于使用单一标签的基线模型。这意味着设计更具包容性的视角感知人工智能模型不仅是实现负责任和符合伦理的人工智能的关键第一步，而且相比传统方法还能获得更优越的结果。