There is a fundamental limitation in the prediction performance that a machine learning model can achieve due to the inevitable uncertainty of the prediction target. In classification problems, this can be characterized by the Bayes error, which is the best achievable error with any classifier. The Bayes error can be used as a criterion to evaluate classifiers with state-of-the-art performance and can be used to detect test set overfitting. We propose a simple and direct Bayes error estimator, where we just take the mean of the labels that show \emph{uncertainty} of the class assignments. Our flexible approach enables us to perform Bayes error estimation even for weakly supervised data. In contrast to others, our method is model-free and even instance-free. Moreover, it has no hyperparameters and gives a more accurate estimate of the Bayes error than several baselines empirically. Experiments using our method suggest that recently proposed deep networks such as the Vision Transformer may have reached, or is about to reach, the Bayes error for benchmark datasets. Finally, we discuss how we can study the inherent difficulty of the acceptance/rejection decision for scientific articles, by estimating the Bayes error of the ICLR papers from 2017 to 2023.

翻译：由于预测目标存在不可避免的不确定性，机器学习模型所能达到的预测性能存在根本性限制。在分类问题中，这一限制可通过贝叶斯误差来表征，即任何分类器所能达到的最佳误差。贝叶斯误差可作为评估最优性能分类器的准则，并可用于检测测试集过拟合。我们提出一种简单直接的贝叶斯误差估计方法，仅需取显示类别分配不确定性的标签均值。该方法具有灵活性，即使在弱监督数据下也能进行贝叶斯误差估计。与其他方法相比，我们的方法无需模型且甚至无需实例，同时无超参数，在经验上能更准确地估计贝叶斯误差。基于该方法进行的实验表明，近期提出的深度网络（如Vision Transformer）可能已接近或即将达到基准数据集的贝叶斯误差。最后，我们通过估计2017年至2023年ICLR论文的贝叶斯误差，探讨如何研究科学论文接受/拒绝决策的固有难度。