With the increasing deployment of machine learning models in many socially-sensitive tasks, there is a growing demand for reliable and trustworthy predictions. One way to accomplish these requirements is to allow a model to abstain from making a prediction when there is a high risk of making an error. This requires adding a selection mechanism to the model, which selects those examples for which the model will provide a prediction. The selective classification framework aims to design a mechanism that balances the fraction of rejected predictions (i.e., the proportion of examples for which the model does not make a prediction) versus the improvement in predictive performance on the selected predictions. Multiple selective classification frameworks exist, most of which rely on deep neural network architectures. However, the empirical evaluation of the existing approaches is still limited to partial comparisons among methods and settings, providing practitioners with little insight into their relative merits. We fill this gap by benchmarking 18 baselines on a diverse set of 44 datasets that includes both image and tabular data. Moreover, there is a mix of binary and multiclass tasks. We evaluate these approaches using several criteria, including selective error rate, empirical coverage, distribution of rejected instance's classes, and performance on out-of-distribution instances. The results indicate that there is not a single clear winner among the surveyed baselines, and the best method depends on the users' objectives.

翻译：随着机器学习模型在诸多社会敏感任务中的广泛应用，对可靠可信预测的需求日益增长。实现这一要求的一种方式是允许模型在预测错误风险较高时放弃做出预测。这需要为模型添加选择机制，筛选出那些模型将提供预测的样本。选择性分类框架旨在设计一种机制，在拒绝预测的比例（即模型不进行预测的样本占比）与被选预测的预测性能提升之间取得平衡。现有多种选择性分类框架，其中大部分基于深度神经网络架构。然而，现有方法在经验评估方面仍局限于方法间与设置间的部分比较，这使得实践者难以洞察其相对优劣。我们通过将18种基线方法在包含图像与表格数据的44个多样化数据集上进行基准测试，填补了这一空白。此外，数据集涵盖二分类与多分类任务。我们采用选择性错误率、经验覆盖率、被拒绝样本类别的分布以及分布外样本的性能等多个标准对这些方法进行评估。结果表明，所调研的基线方法中不存在明确的优胜者，最优方法取决于用户的具体目标。