Current practices regarding data collection for natural language processing on Amazon Mechanical Turk (MTurk) often rely on a combination of studies on data quality and heuristics shared among NLP researchers. However, without considering the perspectives of MTurk workers, these approaches are susceptible to issues regarding workers' rights and poor response quality. We conducted a critical literature review and a survey of MTurk workers aimed at addressing open questions regarding best practices for fair payment, worker privacy, data quality, and considering worker incentives. We found that worker preferences are often at odds with received wisdom among NLP researchers. Surveyed workers preferred reliable, reasonable payments over uncertain, very high payments; reported frequently lying on demographic questions; and expressed frustration at having work rejected with no explanation. We also found that workers view some quality control methods, such as requiring minimum response times or Master's qualifications, as biased and largely ineffective. Based on the survey results, we provide recommendations on how future NLP studies may better account for MTurk workers' experiences in order to respect workers' rights and improve data quality.

翻译：当前，在Amazon Mechanical Turk（MTurk）上进行自然语言处理数据收集的常见实践，往往依赖于数据质量研究以及NLP研究者间共享的经验法则。然而，若不考虑MTurk工人的观点，这些方法容易引发工人权益受损和响应质量低下等问题。我们进行了批判性文献综述，并对MTurk工人展开了一项调查，旨在解答有关公平报酬、工人隐私、数据质量及考虑工人激励因素方面最佳做法的未决问题。研究发现，工人的偏好往往与NLP研究者中普遍认同的见解相悖。受调查工人更倾向于可靠、合理的报酬，而非不确定的高额报酬；他们常声称在人口统计问题中提供虚假信息；并对工作被无解释地拒收表示沮丧。此外，我们还发现，工人认为某些质量控制方法（如要求最短响应时间或大师级资格）存在偏见且基本无效。基于调查结果，我们提出了相关建议，以期未来的NLP研究能更好地纳入MTurk工人的体验，从而尊重工人权益并提升数据质量。