Crowdsourcing has been widely used to efficiently obtain labeled datasets for supervised learning from large numbers of human resources at low cost. However, one of the technical challenges in obtaining high-quality results from crowdsourcing is dealing with the variability and bias caused by the fact that it is humans execute the work, and various studies have addressed this issue to improve the quality by integrating redundantly collected responses. In this study, we focus on the observation bias in crowdsourcing. Variations in the frequency of worker responses and the complexity of tasks occur, which may affect the aggregation results when they are correlated with the quality of the responses. We also propose statistical aggregation methods for crowdsourcing responses that are combined with an observational data bias removal method used in causal inference. Through experiments using both synthetic and real datasets with/without artificially injected spam and colluding workers, we verify that the proposed method improves the aggregation accuracy in the presence of strong observation biases and robustness to both spam and colluding workers.

翻译：众包被广泛应用于以低成本从大量人力资源中高效获取用于监督学习的标注数据集。然而，从众包中获得高质量结果的技术挑战之一是处理由人类执行任务所导致的变异性和偏差，已有诸多研究通过整合冗余收集的响应来解决这一问题以提高质量。在本研究中，我们聚焦于众包中的观测偏差。工作者响应频率和任务复杂性的变化可能在与响应质量相关时影响聚合结果。我们提出了结合因果推断中观测数据偏差去除方法的众包响应统计聚合方法。通过使用合成数据集和真实数据集（含/不含人工注入的垃圾信息及共谋工作者）进行实验，我们验证了所提方法在强观测偏差下能提高聚合精度，并对垃圾信息及共谋工作者具有鲁棒性。