Noise plagues many numerical datasets, where the recorded values in the data may fail to match the true underlying values due to reasons including: erroneous sensors, data entry/processing mistakes, or imperfect human estimates. Here we consider estimating which data values are incorrect along a numerical column. We present a model-agnostic approach that can utilize any regressor (i.e. statistical or machine learning model) which was fit to predict values in this column based on the other variables in the dataset. By accounting for various uncertainties, our approach distinguishes between genuine anomalies and natural data fluctuations, conditioned on the available information in the dataset. We establish theoretical guarantees for our method and show that other approaches like conformal inference struggle to detect errors. We also contribute a new error detection benchmark involving 5 regression datasets with real-world numerical errors (for which the true values are also known). In this benchmark and additional simulation studies, our method identifies incorrect values with better precision/recall than other approaches.

翻译：噪声困扰着众多数值数据集，其中记录的值可能因传感器误差、数据录入/处理错误或人为估计不准确等原因，与真实值不符。本文考虑如何识别数值列中错误的数据值。我们提出一种与模型无关的方法，可利用任何回归器（即统计或机器学习模型），该回归器已根据数据集中的其他变量拟合用于预测该列的值。通过考虑多种不确定性，我们的方法能够基于数据集中的可用信息，区分真实异常与自然数据波动。我们为该方法的理论保证建立了基础，并表明其他方法（如共形推断）难以检测错误。我们还贡献了一个新的错误检测基准，包含5个具有真实数值错误（已知真实值）的回归数据集。在该基准及额外的模拟研究中，我们的方法在查准率/查全率上优于其他方法，更准确地识别出错误值。