We examine the disconnect between scholarship and practice in applying machine learning to trust and safety problems, using misinformation detection as a case study. We systematize literature on automated detection of misinformation across a corpus of 270 well-cited papers in the field. We then examine subsets of papers for data and code availability, design missteps, reproducibility, and generalizability. Our paper corpus includes published work in security, natural language processing, and computational social science. Across these disparate disciplines, we identify common errors in dataset and method design. In general, detection tasks are often meaningfully distinct from the challenges that online services actually face. Datasets and model evaluation are often non-representative of real-world contexts, and evaluation frequently is not independent of model training. Data and code availability is poor. We demonstrate the limitations of current detection methods in a series of three replication studies. Based on the results of these analyses and our literature survey, we offer recommendations for evaluating applications of machine learning to trust and safety problems in general. Our aim is for future work to avoid the pitfalls that we identify.

翻译：我们以虚假信息检测为案例，审视了机器学习在信任与安全问题的学术研究与实践应用之间的脱节。我们对270篇该领域高引用论文构成的文献库中关于虚假信息自动检测的研究进行了系统化梳理。随后，我们考察了部分论文在数据与代码可用性、设计缺陷、可重复性和泛化能力方面的表现。我们的论文库涵盖安全、自然语言处理及计算社会科学等领域已发表的研究成果。在这些不同学科中，我们识别出数据集与方法设计中的常见错误。总体而言，检测任务通常与在线服务实际面临的挑战存在显著差异。数据集与模型评估往往缺乏对真实世界情境的代表性，且评估过程常未独立于模型训练。数据和代码的可用性较差。我们通过三项重复性研究，展示了当前检测方法的局限性。基于这些分析结果及文献调研，我们为评估机器学习在信任与安全问题上的应用提出了通用建议。旨在使未来研究能够规避我们已识别的陷阱。