While the performance of machine learning systems has experienced significant improvement in recent years, relatively little attention has been paid to the fundamental question: to what extent can we improve our models? This paper provides a means of answering this question in the setting of binary classification, which is practical and theoretically supported. We extend a previous work that utilizes soft labels for estimating the Bayes error, the optimal error rate, in two important ways. First, we theoretically investigate the properties of the bias of the hard-label-based estimator discussed in the original work. We reveal that the decay rate of the bias is adaptive to how well the two class-conditional distributions are separated, and it can decay significantly faster than the previous result suggested as the number of hard labels per instance grows. Second, we tackle a more challenging problem setting: estimation with corrupted soft labels. One might be tempted to use calibrated soft labels instead of clean ones. However, we reveal that calibration guarantee is not enough, that is, even perfectly calibrated soft labels can result in a substantially inaccurate estimate. Then, we show that isotonic calibration can provide a statistically consistent estimator under an assumption weaker than that of the previous work. Our method is instance-free, i.e., we do not assume access to any input instances. This feature allows it to be adopted in practical scenarios where the instances are not available due to privacy issues. Experiments with synthetic and real-world datasets show the validity of our methods and theory. The code is available at https://github.com/RyotaUshio/bayes-error-estimation.

翻译：尽管机器学习系统的性能近年来取得了显著提升，但一个根本性问题却鲜受关注：我们的模型究竟能改进到何种程度？本文针对二分类场景提供了一种兼具实用性与理论支撑的答案。我们以两个重要方向拓展了先前利用软标签估计贝叶斯误差（即最优误差率）的研究。首先，我们理论分析了原始工作中基于硬标签估计算法的偏差性质。研究表明，该偏差的衰减率能自适应于两类条件分布间的分离程度，且随每个样本硬标签数量的增长，其衰减速度可显著快于先前结果。其次，我们攻克了更具挑战性的问题设定：利用含噪软标签进行估计。人们或倾向使用经标定的软标签替代清洁标签，但本研究揭示标定保证并不充分——即使是完美标定的软标签也可能导致显著不准确的估计。继而我们证明，在弱于先前工作的假设条件下，等渗回归可提供统计相合的估计量。本方法无需实例访问（即不假设可获取任何输入样本），这一特性使其能被应用于因隐私问题无法获取样本数据的实际场景。基于合成数据与真实数据集的实验验证了方法与理论的有效性。相关代码已开源至 https://github.com/RyotaUshio/bayes-error-estimation。