In supervised learning - for instance in image classification - modern massive datasets are commonly labeled by a crowd of workers. The obtained labels in this crowdsourcing setting are then aggregated for training. The aggregation step generally leverages a per-worker trust score. Yet, such worker-centric approaches discard each task's ambiguity. Some intrinsically ambiguous tasks might even fool expert workers, which could eventually be harmful to the learning step. In a standard supervised learning setting - with one label per task - the Area Under the Margin (AUM) is tailored to identify mislabeled data. We adapt the AUM to identify ambiguous tasks in crowdsourced learning scenarios, introducing the Weighted AUM (WAUM). The WAUM is an average of AUMs weighted by task-dependent scores. We show that the WAUM can help discard ambiguous tasks from the training set, leading to better generalization or calibration performance. We report improvements over existing strategies for learning a crowd, both for simulated settings and for the CIFAR-10H, LabelMe and Music crowdsourced datasets.

翻译：在监督学习（如图像分类）中，现代大规模数据集通常由众包工人标注。众包场景下获得的标签随后被聚合用于训练。聚合步骤通常利用每个工人的信任分数。然而，这种以工人为中心的方法忽略了每个任务的模糊性。某些本质模糊的任务甚至可能误导专家工人，最终对学习步骤不利。在标准监督学习设置（每个任务一个标签）中，边际下面积（AUM）专门用于识别错误标注的数据。我们将AUM应用于众包学习场景中识别模糊任务，引入了加权AUM（WAUM）。WAUM是由任务相关分数加权的AUM平均值。我们证明WAUM可以帮助从训练集中剔除模糊任务，从而提升泛化或校准性能。我们报告了在模拟设置以及CIFAR-10H、LabelMe和Music众包数据集上，该方法相比现有群体学习策略的改进。