Machine learning (ML) and artificial intelligence (AI) systems rely heavily on human-annotated data for training and evaluation. A major challenge in this context is the occurrence of annotation errors, as their effects can degrade model performance. This paper presents a predictive error model trained to detect potential errors in search relevance annotation tasks for three industry-scale ML applications (music streaming, video streaming, and mobile apps). Drawing on real-world data from an extensive search relevance annotation program, we demonstrate that errors can be predicted with moderate model performance (AUC=0.65-0.75) and that model performance generalizes well across applications (i.e., a global, task-agnostic model performs on par with task-specific models). In contrast to past research, which has often focused on predicting annotation labels from task-specific features, our model is trained to predict errors directly from a combination of task features and behavioral features derived from the annotation process, in order to achieve a high degree of generalizability. We demonstrate the usefulness of the model in the context of auditing, where prioritizing tasks with high predicted error probabilities considerably increases the amount of corrected annotation errors (e.g., 40% efficiency gains for the music streaming application). These results highlight that behavioral error detection models can yield considerable improvements in the efficiency and quality of data annotation processes. Our findings reveal critical insights into effective error management in the data annotation process, thereby contributing to the broader field of human-in-the-loop ML.

翻译：机器学习（ML）与人工智能（AI）系统严重依赖人工标注数据进行训练与评估。在此背景下，标注错误的发生是一个主要挑战，其影响可能导致模型性能下降。本文提出了一种预测性错误模型，该模型旨在检测三个工业级机器学习应用（音乐流媒体、视频流媒体与移动应用）中搜索相关性标注任务的潜在错误。基于一项大规模搜索相关性标注项目的真实数据，我们证明错误可以被预测，且模型性能达到中等水平（AUC=0.65-0.75），同时模型性能在不同应用间展现出良好的泛化能力（即一个全局的、任务无关的模型表现与任务专用模型相当）。与以往研究通常侧重于从任务特定特征预测标注标签不同，我们的模型直接结合任务特征与从标注过程中提取的行为特征来预测错误，以实现高度的可泛化性。我们展示了该模型在审计场景中的实用性：优先处理预测错误概率高的任务可显著增加被纠正的标注错误数量（例如，在音乐流媒体应用中实现40%的效率提升）。这些结果表明，行为错误检测模型能够显著提升数据标注过程的效率与质量。我们的发现揭示了数据标注过程中有效错误管理的关键见解，从而为更广泛的人机协同机器学习领域做出贡献。