Accounting for the opinions of all annotators of a dataset is critical for fairness. However, when annotating large datasets, individual annotators will frequently provide thousands of ratings which can lead to fatigue. Additionally, these annotation processes can occur over multiple days which can lead to an inaccurate representation of an annotator's opinion over time. To combat this, we propose to learn a more accurate representation of diverse opinions by utilizing multitask learning in conjunction with loss-based label correction. We show that using our novel formulation, we can cleanly separate agreeing and disagreeing annotations. Furthermore, we demonstrate that this modification can improve prediction performance in a single or multi-annotator setting. Lastly, we show that this method remains robust to additional label noise that is applied to subjective data.

翻译：为保障公平性，需充分考虑数据集中所有标注者的意见。然而，在标注大规模数据集时，个体标注者常需完成数千次评级，导致疲劳效应。此外，标注过程可能持续数日，造成标注者意见随时间推移产生不准确表征。为解决该问题，我们提出通过多任务学习结合基于损失的标签校正方法，学习更精准的多样化意见表征。实验表明，采用新型公式可清晰分离一致性与分歧性标注。进一步证明，该改进能在单标注者或多标注者场景中提升预测性能。最后，该方法对主观数据中引入的额外标签噪声仍保持鲁棒性。