Pilot studies are an essential cornerstone of the design of crowdsourcing campaigns, yet they are often only mentioned in passing in the scholarly literature. A lack of details surrounding pilot studies in crowdsourcing research hinders the replication of studies and the reproduction of findings, stalling potential scientific advances. We conducted a systematic literature review on the current state of pilot study reporting at the intersection of crowdsourcing and HCI research. Our review of ten years of literature included 171 articles published in the proceedings of the Conference on Human Computation and Crowdsourcing (AAAI HCOMP) and the ACM Digital Library. We found that pilot studies in crowdsourcing research (i.e., crowd pilot studies) are often under-reported in the literature. Important details, such as the number of workers and rewards to workers, are often not reported. On the basis of our findings, we reflect on the current state of practice and formulate a set of best practice guidelines for reporting crowd pilot studies in crowdsourcing research. We also provide implications for the design of crowdsourcing platforms and make practical suggestions for supporting crowd pilot study reporting.

翻译：试点研究是众包活动设计的基石，然而在学术文献中常被一笔带过。众包研究中试点研究细节的匮乏阻碍了研究的可重复性与结果的可再现性，从而制约了潜在的科学进展。我们针对众包与人机交互交叉领域试点研究报告的现状开展了系统性文献综述。通过对十年间文献的梳理，共涵盖发表于人类计算与众包会议及ACM数字图书馆的171篇论文。研究发现，众包研究中的试点研究（即众包试点研究）在文献中往往被低度报告，诸如工人数量及工人报酬等关键细节常被遗漏。基于研究结果，我们反思当前实践现状，并制定了一套针对众包研究中众包试点研究报告的最佳实践指南。此外，我们还为众包平台的设计提供了启示，并就支持众包试点研究报告提出了切实建议。