We propose a new Reject Option Classification technique to identify and remove regions of uncertainty in the decision space for a given neural classifier and dataset. Such existing formulations employ a learned rejection (remove)/selection (keep) function and require either a known cost for rejecting examples or strong constraints on the accuracy or coverage of the selected examples. We consider an alternative formulation by instead analyzing the complementary reject region and employing a validation set to learn per-class softmax thresholds. The goal is to maximize the accuracy of the selected examples subject to a natural randomness allowance on the rejected examples (rejecting more incorrect than correct predictions). We provide results showing the benefits of the proposed method over na\"ively thresholding calibrated/uncalibrated softmax scores with 2-D points, imagery, and text classification datasets using state-of-the-art pretrained models. Source code is available at https://github.com/osu-cvl/learning-idk.

翻译：我们提出了一种新的拒绝选项分类技术，用于识别并移除给定神经分类器和数据集决策空间中的不确定性区域。现有此类方法采用学习得到的拒绝（移除）/选择（保留）函数，且需要已知的拒绝样本成本或对所选样本的准确率/覆盖率的严格约束。我们通过分析互补的拒绝区域并利用验证集学习每个类别的softmax阈值，提出了一种替代性方案。其目标是在对拒绝样本设置自然随机性允许范围（即拒绝更多错误预测而非正确预测）的前提下，最大化所选样本的准确率。我们提供的实验结果表明，在二维点、图像和文本分类数据集上，相较于直接对校准/未校准的softmax分数进行阈值化处理，该方法在使用最先进预训练模型时具有显著优势。源代码已开放于 https://github.com/osu-cvl/learning-idk。