Subjective NLP tasks usually rely on human annotations provided by multiple annotators, whose judgments may vary due to their diverse backgrounds and life experiences. Traditional methods often aggregate multiple annotations into a single ground truth, disregarding the diversity in perspectives that arises from annotator disagreement. In this preliminary study, we examine the effect of including multiple annotations on model accuracy in classification. Our methodology investigates the performance of perspective-aware classification models in stance detection task and further inspects if annotator disagreement affects the model confidence. The results show that multi-perspective approach yields better classification performance outperforming the baseline which uses the single label. This entails that designing more inclusive perspective-aware AI models is not only an essential first step in implementing responsible and ethical AI, but it can also achieve superior results than using the traditional approaches.

翻译：主观性自然语言处理任务通常依赖于多位标注者提供的人工标注，由于标注者背景和生活经历的多样性，其判断可能存在差异。传统方法常将多个标注聚合为单一标准答案，忽视了因标注者分歧而产生的视角多样性。在本初步研究中，我们探讨了在分类任务中包含多标注对模型准确性的影响。我们的方法研究了视角感知分类模型在立场检测任务中的表现，并进一步检验了标注者分歧是否影响模型置信度。结果表明，多视角方法相比使用单一标签的基线模型取得了更好的分类性能。这意味着设计更具包容性的视角感知人工智能模型不仅是实现负责任和伦理人工智能的关键第一步，还能获得优于传统方法的结果。