Large-scale datasets are important for the development of deep learning models. Such datasets usually require a heavy workload of annotations, which are extremely time-consuming and expensive. To accelerate the annotation procedure, multiple annotators may be employed to label different subsets of the data. However, the inconsistency and bias among different annotators are harmful to the model training, especially for qualitative and subjective tasks.To address this challenge, in this paper, we propose a novel contrastive regression framework to address the disjoint annotations problem, where each sample is labeled by only one annotator and multiple annotators work on disjoint subsets of the data. To take account of both the intra-annotator consistency and inter-annotator inconsistency, two strategies are employed.Firstly, a contrastive-based loss is applied to learn the relative ranking among different samples of the same annotator, with the assumption that the ranking of samples from the same annotator is unanimous. Secondly, we apply the gradient reversal layer to learn robust representations that are invariant to different annotators. Experiments on the facial expression prediction task, as well as the image quality assessment task, verify the effectiveness of our proposed framework.

翻译：大规模数据集对深度学习模型的发展至关重要，但此类数据集通常需要繁重的标注工作，耗时且成本极高。为加速标注流程，可能雇佣多位标注员对数据的不同子集进行标注，然而不同标注员之间的不一致性和偏差会对模型训练产生负面影响，尤其是在定性与主观性任务中。针对这一挑战，本文提出了一种新颖的对比回归框架，用于解决分离标注问题，即每个样本仅由一位标注员标注，且多位标注员处理互不重叠的数据子集。为兼顾标注员内部一致性及标注员间不一致性，我们采用两种策略：首先，应用基于对比的损失函数学习同一标注员内不同样本的相对排序，假设同一标注员对样本的排序具有一致性；其次，利用梯度反转层学习对各标注员不变性的鲁棒表征。在面部表情预测任务及图像质量评估任务上的实验验证了所提框架的有效性。