In order to trust the predictions of a machine learning algorithm, it is necessary to understand the factors that contribute to those predictions. In the case of probabilistic and uncertainty-aware models, it is necessary to understand not only the reasons for the predictions themselves, but also the reasons for the model's level of confidence in those predictions. In this paper, we show how existing methods in explainability can be extended to uncertainty-aware models and how such extensions can be used to understand the sources of uncertainty in a model's predictive distribution. In particular, by adapting permutation feature importance, partial dependence plots, and individual conditional expectation plots, we demonstrate that novel insights into model behaviour may be obtained and that these methods can be used to measure the impact of features on both the entropy of the predictive distribution and the log-likelihood of the ground truth labels under that distribution. With experiments using both synthetic and real-world data, we demonstrate the utility of these approaches to understand both the sources of uncertainty and their impact on model performance.

翻译：为了信任机器学习算法的预测，有必要理解影响这些预测的因素。对于概率性和不确定性感知模型，不仅需要理解预测本身的原因，还需要理解模型对这些预测置信度的原因。本文展示了如何将可解释性领域的现有方法扩展至不确定性感知模型，以及如何利用此类扩展来理解模型预测分布中不确定性的来源。具体而言，通过改进置换特征重要性、部分依赖图和个体条件期望图，我们证明了可以获得对模型行为的新见解，并且这些方法可用于度量特征对预测分布熵以及该分布下真实标签对数似然的影响。通过使用合成数据和真实世界数据的实验，我们证明了这些方法在理解不确定性来源及其对模型性能影响方面的实用性。