Crowdsourcing is a common approach to rapidly annotate large volumes of data in machine learning applications. Typically, crowd workers are compensated with a flat rate based on an estimated completion time to meet a target hourly wage. Unfortunately, prior work has shown that variability in completion times among crowd workers led to overpayment by 168% in one case, and underpayment by 16% in another. However, by setting a time limit for task completion, it is possible to manage the risk of overpaying or underpaying while still facilitating flat rate payments. In this paper, we present an analysis of the impact of a time limit on crowd worker performance and satisfaction. We conducted a human study with a maximum view time for a crowdsourced image classification task. We find that the impact on overall crowd worker performance diminishes as view time increases. Despite some images being challenging under time limits, a consensus algorithm remains effective at preserving data quality and filters images needing more time. Additionally, crowd workers' consistent performance throughout the time-limited task indicates sustained effort, and their psychometric questionnaire scores show they prefer shorter limits. Based on our findings, we recommend implementing task time limits as a practical approach to making compensation more equitable and predictable.

翻译：众包是机器学习应用中快速标注大规模数据的常用方法。通常，众包工作者会基于预估完成时间获得固定报酬，以达到目标时薪。然而，已有研究表明，众包工作者完成时间的差异性曾导致报酬超额支付达168%，而在另一案例中报酬不足支付达16%。通过设定任务完成的时间限制，可以在维持固定报酬支付的同时，管理超额支付或不足支付的风险。本文分析了时间限制对众包工作者表现与满意度的影响。我们通过一项设定最长查看时间的众包图像分类任务开展了人工实验。研究发现，随着查看时间增加，时间限制对众包工作者整体表现的影响逐渐减弱。尽管部分图像在时间限制下具有挑战性，共识算法仍能有效保持数据质量，并筛选出需要更长时间处理的图像。此外，众包工作者在限时任务中表现稳定，表明其持续投入的努力；心理测量问卷得分显示他们更倾向于较短的时间限制。基于研究结果，我们建议实施任务时间限制作为一种实用方法，使报酬分配更公平且可预测。