Classification is a fundamental task in many applications on which data-driven methods have shown outstanding performances. However, it is challenging to determine whether such methods have achieved the optimal performance. This is mainly because the best achievable performance is typically unknown and hence, effectively estimating it is of prime importance. In this paper, we consider binary classification problems and we propose an estimator for the false positive rate (FPR) of the Bayes classifier, that is, the optimal classifier with respect to accuracy, from a given dataset. Our method utilizes soft labels, or real-valued labels, which are gaining significant traction thanks to their properties. We thoroughly examine various theoretical properties of our estimator, including its consistency, unbiasedness, rate of convergence, and variance. To enhance the versatility of our estimator beyond soft labels, we also consider noisy labels, which encompass binary labels. For noisy labels, we develop effective FPR estimators by leveraging a denoising technique and the Nadaraya-Watson estimator. Due to the symmetry of the problem, our results can be readily applied to estimate the false negative rate of the Bayes classifier.

翻译：分类是众多应用中的基本任务，数据驱动方法在此领域展现出卓越性能。然而，判断此类方法是否达到最优性能颇具挑战性。这主要源于最优可达性能通常未知，因此对其进行有效估计至关重要。本文针对二分类问题，提出一种基于给定数据集估计贝叶斯分类器（即精度最优分类器）误报率（FPR）的估计方法。该方法利用软标签（即实值标签），此类标签因其特性正受到广泛关注。我们深入分析了所提估计器的多种理论性质，包括一致性、无偏性、收敛速率及方差。为提升估计器在软标签之外的普适性，我们进一步考虑包含二值标签的噪声标签情形。针对噪声标签，我们借助去噪技术和Nadaraya-Watson估计器开发了有效的FPR估计方法。基于问题的对称性，本研究成果可便捷应用于贝叶斯分类器漏报率的估计。