Noise plagues many numerical datasets, where the recorded values in the data may fail to match the true underlying values due to reasons including: erroneous sensors, data entry/processing mistakes, or imperfect human estimates. Here we consider estimating \emph{which} data values are incorrect along a numerical column. We present a model-agnostic approach that can utilize \emph{any} regressor (i.e.\ statistical or machine learning model) which was fit to predict values in this column based on the other variables in the dataset. By accounting for various uncertainties, our approach distinguishes between genuine anomalies and natural data fluctuations, conditioned on the available information in the dataset. We establish theoretical guarantees for our method and show that other approaches like conformal inference struggle to detect errors. We also contribute a new error detection benchmark involving 5 regression datasets with real-world numerical errors (for which the true values are also known). In this benchmark and additional simulation studies, our method identifies incorrect values with better precision/recall than other approaches.

翻译：噪声困扰着许多数值数据集，由于传感器错误、数据录入/处理错误或人为估算不精确等原因，数据中的记录值可能无法匹配真实值。本文针对数值列中的数据错误进行识别研究。我们提出了一种与模型无关的方法，该方法可利用任意基于数据集中其他变量对该列数值进行预测的回归模型（即统计或机器学习模型）。通过考虑多种不确定性，我们的方法能够根据数据集中的可用信息，区分真正的异常值与自然的数据波动。我们为该方法建立了理论保障，并表明其他方法（如共形推断）在错误检测方面表现不佳。我们还贡献了一个新的错误检测基准，包含5个具有真实世界数值错误（同时已知真实值）的回归数据集。在该基准测试及额外的模拟研究中，我们的方法在识别错误值方面的精确率/召回率均优于其他方法。