In supervised learning - for instance in image classification - modern massive datasets are commonly labeled by a crowd of workers. The obtained labels in this crowdsourcing setting are then aggregated for training, generally leveraging a per-worker trust score. Yet, such workers oriented approaches discard the tasks' ambiguity. Ambiguous tasks might fool expert workers, which is often harmful for the learning step. In standard supervised learning settings - with one label per task - the Area Under the Margin (AUM) was tailored to identify mislabeled data. We adapt the AUM to identify ambiguous tasks in crowdsourced learning scenarios, introducing the Weighted Areas Under the Margin (WAUM). The WAUM is an average of AUMs weighted according to task-dependent scores. We show that the WAUM can help discarding ambiguous tasks from the training set, leading to better generalization performance. We report improvements over existing strategies for learning with a crowd, both on simulated settings, and on real datasets such as CIFAR-10H (a crowdsourced dataset with a high number of answered labels),LabelMe and Music (two datasets with few answered votes).

翻译：在监督学习中（例如图像分类），现代大规模数据集通常由众包工人进行标注。在这种众包场景中，获得的标签随后被聚合用于训练，通常利用每个工人的可信度评分。然而，这种面向工人的方法忽略了任务的模糊性。模糊任务可能误导专家级工人，这对学习步骤往往有害。在标准监督学习场景（每个任务一个标签）中，边际下面积（AUM）被设计用于识别错误标注的数据。我们将AUM调整为识别众包学习场景中的模糊任务，引入了加权边际下面积（WAUM）。WAUM是根据任务相关评分加权的AUM平均值。我们证明，WAUM有助于从训练集中剔除模糊任务，从而获得更好的泛化性能。我们报告了相较于现有众包学习策略的提升，无论是在模拟场景还是真实数据集上，如CIFAR-10H（一个带有大量回答标签的众包数据集）、LabelMe和Music（两个投票数量较少的数据集）。