While most machine learning models can provide confidence in their predictions, confidence is insufficient to understand a prediction's reliability. For instance, the model may have a low confidence prediction if the input is not well-represented in the training dataset or if the input is inherently ambiguous. In this work, we investigate the relationship between how atypical(rare) a sample or a class is and the reliability of a model's predictions. We first demonstrate that atypicality is strongly related to miscalibration and accuracy. In particular, we empirically show that predictions for atypical inputs or atypical classes are more overconfident and have lower accuracy. Using these insights, we show incorporating atypicality improves uncertainty quantification and model performance for discriminative neural networks and large language models. In a case study, we show that using atypicality improves the performance of a skin lesion classifier across different skin tone groups without having access to the group attributes. Overall, we propose that models should use not only confidence but also atypicality to improve uncertainty quantification and performance. Our results demonstrate that simple post-hoc atypicality estimators can provide significant value.

翻译：尽管大多数机器学习模型能够提供预测置信度，但仅凭置信度不足以理解预测的可靠性。例如，当输入在训练数据集中代表性不足或输入本身存在模糊性时，模型可能给出低置信度预测。本研究探讨了样本或类别的非典型性（罕见程度）与模型预测可靠性之间的关系。我们首先证明非典型性与校准误差和准确率高度相关。具体而言，实验表明：非典型输入或非典型类别的预测更易出现过度自信现象，且准确率较低。基于这些发现，我们证明融入非典型性可改善判别式神经网络和大语言模型的不确定性量化与模型性能。在案例研究中，我们展示了在无需获取群体属性信息的情况下，利用非典型性可提升皮肤病变分类器在不同肤色群体中的表现。总体而言，我们提出模型不仅应使用置信度，还应考虑非典型性以提升不确定性量化与性能。实验结果表明，简单的后验非典型性估计器具有显著价值。