Convolutional neural networks (CNNs) are extremely popular and effective for image classification tasks but tend to be overly confident in their predictions. Various works have sought to quantify uncertainty associated with these models, detect out-of-distribution (OOD) inputs, or identify anomalous regions in an image, but limited work has sought to develop a holistic approach that can accurately estimate perception model confidence across various sources of uncertainty. We develop a probabilistic and reconstruction-based competency estimation (PaRCE) method and compare it to existing approaches for uncertainty quantification and OOD detection. We find that our method can best distinguish between correctly classified, misclassified, and OOD samples with anomalous regions, as well as between samples with visual image modifications resulting in high, medium, and low prediction accuracy. We describe how to extend our approach for anomaly localization tasks and demonstrate the ability of our approach to distinguish between regions in an image that are familiar to the perception model from those that are unfamiliar. We find that our method generates interpretable scores that most reliably capture a holistic notion of perception model confidence.

翻译：卷积神经网络（CNN）在图像分类任务中极为流行且高效，但其预测往往存在过度自信的问题。现有研究多集中于量化模型不确定性、检测分布外（OOD）输入或识别图像异常区域，而鲜有工作致力于构建能够综合评估感知模型在不同不确定性来源下置信度的整体性方法。本文提出一种基于概率与重构的能力估计（PaRCE）方法，并与现有不确定性量化和OOD检测方法进行对比。实验表明，本方法能有效区分正确分类样本、误分类样本及含异常区域的OOD样本，并能识别因视觉图像修改导致预测准确率高、中、低的不同样本。我们进一步阐述了如何将本方法扩展至异常定位任务，并证明其能够有效区分图像中感知模型熟悉与不熟悉的区域。结果表明，本方法生成的解释性评分能够最可靠地反映感知模型置信度的整体性概念。