We consider the problem of estimating the false-/ true-positive-rate (FPR/TPR) for a binary classification model when there are incorrect labels (label noise) in the validation set. Our motivating application is fraud prevention where accurate estimates of FPR are critical to preserving the experience for good customers, and where label noise is highly asymmetric. Existing methods seek to minimize the total error in the cleaning process - to avoid cleaning examples that are not noise, and to ensure cleaning of examples that are. This is an important measure of accuracy but insufficient to guarantee good estimates of the true FPR or TPR for a model, and we show that using the model to directly clean its own validation data leads to underestimates even if total error is low. This indicates a need for researchers to pursue methods that not only reduce total error but also seek to de-correlate cleaning error with model scores.

翻译：我们考虑在验证集中存在错误标签（标签噪声）时，二分类模型的误报率/真阳性率（FPR/TPR）估计问题。本研究的核心应用场景是欺诈预防——该领域对FPR的准确估计至关重要，以保障优质客户体验，且标签噪声具有高度不对称性。现有方法致力于最小化清洗过程中的总误差：既避免清洗非噪声样本，又确保噪声样本被完全清除。这虽然是重要的准确度指标，但不足以保证模型真实FPR/TPR的可靠估计。我们证明：即使总误差较低，直接使用模型清洗自身验证数据仍会导致估计值偏低。这表明研究者需要探索不仅能降低总误差，更能实现清洗误差与模型分数去相关的方法。