Crowdsourcing platforms use various truth discovery algorithms to aggregate annotations from multiple labelers. In an online setting, however, the main challenge is to decide whether to ask for more annotations for each item to efficiently trade off cost (i.e., the number of annotations) for quality of the aggregated annotations. In this paper, we propose a novel approach for general complex annotation (such as bounding boxes and taxonomy paths), that works in an online crowdsourcing setting. We prove that the expected average similarity of a labeler is linear in their accuracy \emph{conditional on the reported label}. This enables us to infer reported label accuracy in a broad range of scenarios. We conduct extensive evaluations on real-world crowdsourcing data from Meta and show the effectiveness of our proposed online algorithms in improving the cost-quality trade-off.

翻译：众包平台利用多种真相发现算法聚合来自多个标注者的标注结果。然而，在在线设置下，主要挑战在于决定是否为每个项目请求更多标注，以有效权衡成本（即标注数量）与聚合标注质量。本文针对一般性复杂标注（如边界框和分类路径）提出了一种新颖方法，适用于在线众包环境。我们证明了标注者的期望平均相似度与其准确性成线性关系，且该相关性以所报告的标签为条件。这使我们能够在广泛场景下推断所报告标签的准确性。我们基于Meta的真实众包数据进行了大量评估，结果表明所提出的在线算法在改善成本-质量权衡方面具有有效性。