Current practices regarding data collection for natural language processing on Amazon Mechanical Turk (MTurk) often rely on a combination of studies on data quality and heuristics shared among NLP researchers. However, without considering the perspectives of MTurk workers, these approaches are susceptible to issues regarding workers' rights and poor response quality. We conducted a critical literature review and a survey of MTurk workers aimed at addressing open questions regarding best practices for fair payment, worker privacy, data quality, and considering worker incentives. We found that worker preferences are often at odds with received wisdom among NLP researchers. Surveyed workers preferred reliable, reasonable payments over uncertain, very high payments; reported frequently lying on demographic questions; and expressed frustration at having work rejected with no explanation. We also found that workers view some quality control methods, such as requiring minimum response times or Master's qualifications, as biased and largely ineffective. Based on the survey results, we provide recommendations on how future NLP studies may better account for MTurk workers' experiences in order to respect workers' rights and improve data quality.

翻译：当前，自然语言处理领域在Amazon Mechanical Turk（MTurk）上进行数据收集的常见实践，往往依赖于关于数据质量的研究成果以及NLP研究人员之间共享的启发式方法。然而，若不考虑MTurk工作者的视角，这些方法容易引发工作者权益受损和响应质量低下等问题。我们开展了一项批判性文献综述，并对MTurk工作者进行了调研，旨在回答关于公平报酬、工作者隐私、数据质量以及考虑工作者激励机制等最佳实践中的开放性问题。研究发现，工作者的偏好常常与NLP研究人员中普遍认同的观点相悖。接受调研的工作者更倾向于稳定合理的报酬，而非不确定的高额报酬；他们经常在人口统计问题上撒谎；并对工作被无解释地退回表示沮丧。我们还发现，工作者认为某些质量控制方法（例如要求最短响应时间或掌握Master资格）存在偏见且基本无效。基于调研结果，我们提出了相关建议，以指导未来的NLP研究如何更好地考虑MTurk工作者的体验，从而尊重工作者的权益并提升数据质量。