Human data annotation is critical in shaping the quality of machine learning (ML) and artificial intelligence (AI) systems. One significant challenge in this context is posed by annotation errors, as their effects can degrade the performance of ML models. This paper presents a predictive error model trained to detect potential errors in search relevance annotation tasks for three industry-scale ML applications (music streaming, video streaming, and mobile apps) and assesses its potential to enhance the quality and efficiency of the data annotation process. Drawing on real-world data from an extensive search relevance annotation program, we illustrate that errors can be predicted with moderate model performance (AUC=0.65-0.75) and that model performance generalizes well across applications (i.e., a global, task-agnostic model performs on par with task-specific models). We present model explainability analyses to identify which types of features are the main drivers of predictive performance. Additionally, we demonstrate the usefulness of the model in the context of auditing, where prioritizing tasks with high predicted error probabilities considerably increases the amount of corrected annotation errors (e.g., 40% efficiency gains for the music streaming application). These results underscore that automated error detection models can yield considerable improvements in the efficiency and quality of data annotation processes. Thus, our findings reveal critical insights into effective error management in the data annotation process, thereby contributing to the broader field of human-in-the-loop ML.

翻译：人类数据标注对于塑造机器学习（ML）和人工智能（AI）系统的质量至关重要。在此背景下，标注错误构成一项重大挑战，因其影响可能降低机器学习模型的性能。本文提出一种预测性错误模型，该模型针对三种工业级机器学习应用（音乐流媒体、视频流媒体和移动应用）中的搜索相关性标注任务，训练用于检测潜在错误，并评估其提升数据标注流程质量与效率的潜力。基于一个大规模搜索相关性标注项目中的真实数据，我们展示了错误预测可达到中等模型性能（AUC=0.65-0.75），且模型性能在各应用间具有良好的泛化能力（即一个全局的、任务无关的模型与各任务专属模型表现相当）。我们通过模型可解释性分析，识别出哪些特征类型是预测性能的主要驱动因素。此外，我们验证了该模型在审计场景中的实用性：通过优先处理预测错误概率较高的任务，可显著增加被纠正的标注错误数量（例如，在音乐流媒体应用中实现40%的效率提升）。这些结果强调，自动化错误检测模型能够显著提升数据标注流程的效率与质量。因此，我们的研究揭示了数据标注过程中有效错误管理的关键见解，从而为人机协作机器学习这一更广泛的领域做出贡献。