Reliably labelling data typically requires annotations from multiple human workers. However, humans are far from being perfect. Hence, it is a common practice to aggregate labels gathered from multiple annotators to make a more confident estimate of the true label. Among many aggregation methods, the simple and well known Majority Vote (MV) selects the class label polling the highest number of votes. However, despite its importance, the optimality of MV's label aggregation has not been extensively studied. We address this gap in our work by characterising the conditions under which MV achieves the theoretically optimal lower bound on label estimation error. Our results capture the tolerable limits on annotation noise under which MV can optimally recover labels for a given class distribution. This certificate of optimality provides a more principled approach to model selection for label aggregation as an alternative to otherwise inefficient practices that sometimes include higher experts, gold labels, etc., that are all marred by the same human uncertainty despite huge time and monetary costs. Experiments on both synthetic and real world data corroborate our theoretical findings.

翻译：可靠的数据标注通常需要多位人工标注者的注释。然而，人类远非完美。因此，聚合来自多位标注者的标签以对真实标签做出更有信心的估计，已成为一种常见做法。在众多聚合方法中，简单且广为人知的多数投票法选择获得最高票数的类别标签。然而，尽管其重要性不言而喻，多数投票在标签聚合方面的最优性尚未得到深入研究。我们在工作中通过刻画多数投票达到标签估计误差理论最优下界的条件，来填补这一空白。我们的结果揭示了在给定类别分布下，多数投票能够最优恢复标签所能容忍的标注噪声极限。这一最优性证明为标签聚合的模型选择提供了一种更具原则性的方法，替代了那些有时包含更高级专家、黄金标准标签等低效做法——尽管耗费巨大时间和金钱成本，这些做法同样受制于相同的人类不确定性。在合成数据和真实数据上的实验均证实了我们的理论发现。