We examine the disconnect between scholarship and practice in applying machine learning to trust and safety problems, using misinformation detection as a case study. We systematize literature on automated detection of misinformation across a corpus of 270 well-cited papers in the field. We then examine subsets of papers for data and code availability, design missteps, reproducibility, and generalizability. We find significant shortcomings in the literature that call into question claimed performance and practicality. Detection tasks are often meaningfully distinct from the challenges that online services actually face. Datasets and model evaluation are often non-representative of real-world contexts, and evaluation frequently is not independent of model training. Data and code availability is poor. Models do not generalize well to out-of-domain data. Based on these results, we offer recommendations for evaluating machine learning applications to trust and safety problems. Our aim is for future work to avoid the pitfalls that we identify.

翻译：我们以虚假信息检测为案例，审视了机器学习在信任与安全问题应用中学术研究与实际实践之间的脱节。通过对该领域270篇高引用论文的系统化梳理，我们分析了自动化虚假信息检测的相关文献。随后，我们针对其中部分论文的数据与代码可获取性、设计缺陷、可重复性及泛化能力进行了考察。研究发现，现有文献存在显著缺陷，导致其声称的性能与实际可行性存疑：检测任务通常与在线服务面临的实际挑战存在显著差异；数据集与模型评估往往无法代表真实世界场景，且评估过程常与模型训练未实现独立分离；数据和代码的可获取性较差；模型在跨领域数据上的泛化能力不足。基于这些发现，我们提出了评估机器学习在信任与安全问题中应用的建议，旨在推动未来研究规避我们所识别的这些陷阱。