Crowdsourcing is the outsourcing of tasks to a crowd of contributors on a dedicated platform. The crowd on these platforms is very diversified and includes various profiles of contributors which generates data of uneven quality. However, majority voting, which is the aggregating method commonly used in platforms, gives equal weight to each contribution. To overcome this problem, we propose a method, MONITOR, which estimates the contributor's profile and aggregates the collected data by taking into account their possible imperfections thanks to the theory of belief functions. To do so, MONITOR starts by estimating the profile of the contributor through his qualification for the task and his behavior.Crowdsourcing campaigns have been carried out to collect the necessary data to test MONITOR on real data in order to compare it to existing approaches. The results of the experiments show that thanks to the use of the MONITOR method, we obtain a better rate of correct answer after aggregation of the contributions compared to the majority voting. Our contributions in this article are for the first time the proposal of a model that takes into account both the qualification of the contributor and his behavior in the estimation of his profile. For the second one, the weakening and the aggregation of the answers according to the estimated profiles.
翻译:众包是将任务外包给专用平台上的贡献者群体。这些平台上的群体高度多样化,包含各种类型的贡献者,从而导致生成的数据质量参差不齐。然而,平台常用的聚合方法——多数投票——赋予每项贡献相同的权重。为解决这一问题,我们提出了一种名为MONITOR的方法,该方法借助信念函数理论,通过考虑贡献者可能的缺陷来评估其画像,并聚合收集的数据。具体而言,MONITOR首先通过贡献者对任务的资质及其行为来评估其画像。我们开展众包活动以收集必要数据,在真实数据上测试MONITOR,并将其与现有方法进行比较。实验结果表明,采用MONITOR方法后,与多数投票相比,我们在贡献聚合后获得了更高的正确答案率。本文的贡献在于:首次提出一个同时考虑贡献者资质与行为来评估其画像的模型;其次,根据评估画像对答案进行弱化与聚合。