Uncertainty is a key feature of any machine learning model and is particularly important in neural networks, which tend to be overconfident. This overconfidence is worrying under distribution shifts, where the model performance silently degrades as the data distribution diverges from the training data distribution. Uncertainty estimation offers a solution to overconfident models, communicating when the output should (not) be trusted. Although methods for uncertainty estimation have been developed, they have not been explicitly linked to the field of explainable artificial intelligence (XAI). Furthermore, literature in operations research ignores the actionability component of uncertainty estimation and does not consider distribution shifts. This work proposes a general uncertainty framework, with contributions being threefold: (i) uncertainty estimation in ML models is positioned as an XAI technique, giving local and model-specific explanations; (ii) classification with rejection is used to reduce misclassifications by bringing a human expert in the loop for uncertain observations; (iii) the framework is applied to a case study on neural networks in educational data mining subject to distribution shifts. Uncertainty as XAI improves the model's trustworthiness in downstream decision-making tasks, giving rise to more actionable and robust machine learning systems in operations research.

翻译：不确定性是任何机器学习模型的关键特征，在神经网络中尤为重要——这类模型往往表现出过度自信。当数据分布偏离训练数据分布时，模型性能会悄然下降，这种过度自信在分布偏移下令人担忧。不确定性估计为过度自信的模型提供了解决方案，可传达模型输出何时（不）应被信任。尽管已有诸多不确定性估计方法，但它们尚未与可解释人工智能（XAI）领域建立明确联系。此外，运筹学文献忽视了不确定性估计的可行动性要素，亦未考虑分布偏移问题。本文提出一个通用不确定性框架，贡献体现在三个方面：（i）将机器学习模型中的不确定性估计定位为一种XAI技术，提供局部且模型特定的解释；（ii）通过引入人类专家参与不确定观测的拒绝分类机制来减少误分类；（iii）将该框架应用于教育数据挖掘中受分布偏移影响的神经网络案例研究。作为XAI的不确定性估计提升了模型在下游决策任务中的可信度，为运筹学领域催生更具可行动性和鲁棒性的机器学习系统。