Crowdsourcing has emerged as an effective platform for labeling large amounts of data in a cost- and time-efficient manner. Most previous work has focused on designing an efficient algorithm to recover only the ground-truth labels of the data. In this paper, we consider multi-choice crowdsourcing tasks with the goal of recovering not only the ground truth, but also the most confusing answer and the confusion probability. The most confusing answer provides useful information about the task by revealing the most plausible answer other than the ground truth and how plausible it is. To theoretically analyze such scenarios, we propose a model in which there are the top two plausible answers for each task, distinguished from the rest of the choices. Task difficulty is quantified by the probability of confusion between the top two, and worker reliability is quantified by the probability of giving an answer among the top two. Under this model, we propose a two-stage inference algorithm to infer both the top two answers and the confusion probability. We show that our algorithm achieves the minimax optimal convergence rate. We conduct both synthetic and real data experiments and demonstrate that our algorithm outperforms other recent algorithms. We also show the applicability of our algorithms in inferring the difficulty of tasks and in training neural networks with top-two soft labels.

翻译：众包已成为一种以低成本和高效率方式标注大量数据的有效平台。以往研究大多聚焦于设计高效算法仅恢复数据的真实标签。本文考虑多选项众包任务，目标不仅在于恢复真实标签，还在于恢复最易混淆的答案及其混淆概率。最易混淆的答案揭示了任务中除真实标签外最可能的答案及其可能性，从而提供有用信息。为从理论上分析此类场景，我们提出一个模型，其中每个任务存在两个最可能的答案，与其他选项区分开来。任务难度通过前二答案之间的混淆概率来量化，而工人可靠性则通过给出前二答案之一的概率来量化。在此模型下，我们提出两阶段推断算法，同时推断前二答案和混淆概率。我们证明该算法达到了极小化最优收敛速率。通过合成数据和真实数据实验，我们展示了该算法优于其他近期算法。我们还证明了算法在推断任务难度及使用前二软标签训练神经网络方面的适用性。