Crowdsourcing system has emerged as an effective platform for labeling data with relatively low cost by using non-expert workers. Inferring correct labels from multiple noisy answers on data, however, has been a challenging problem, since the quality of the answers varies widely across tasks and workers. Many existing works have assumed that there is a fixed ordering of workers in terms of their skill levels, and focused on estimating worker skills to aggregate the answers from workers with different weights. In practice, however, the worker skill changes widely across tasks, especially when the tasks are heterogeneous. In this paper, we consider a new model, called $d$-type specialization model, in which each task and worker has its own (unknown) type and the reliability of each worker can vary in the type of a given task and that of a worker. We allow that the number $d$ of types can scale in the number of tasks. In this model, we characterize the optimal sample complexity to correctly infer the labels within any given accuracy, and propose label inference algorithms achieving the order-wise optimal limit even when the types of tasks or those of workers are unknown. We conduct experiments both on synthetic and real datasets, and show that our algorithm outperforms the existing algorithms developed based on more strict model assumptions.

翻译：众包系统已成为通过非专业工作者以较低成本进行数据标注的有效平台。然而，从多个含噪数据答案中推断正确标签一直是一个具有挑战性的问题，因为答案质量在不同任务和工作者之间存在显著差异。许多现有工作假设工作者技能存在固定排序，并专注于估计工作者技能以对不同权重的答案进行聚合。然而在实际中，工作者技能在不同任务间变化很大，尤其是在任务异质的情况下。本文提出一种称为$d$型专业化模型的新模型，其中每个任务和工作者具有自身（未知）类型，且每个工作者的可靠性可能因给定任务类型和工作者类型而异。我们允许类型数量$d$随任务数量扩展。在该模型中，我们刻画了在给定精度下正确推断标签的最优样本复杂度，并提出了标签推断算法——即使在工作-任务类型未知的情况下也能达到阶最优极限。我们在合成数据集和真实数据集上进行了实验，表明所提出的算法优于基于更严格模型假设的现有算法。