Uncertainty representation and quantification are paramount in machine learning and constitute an important prerequisite for safety-critical applications. In this paper, we propose novel measures for the quantification of aleatoric and epistemic uncertainty based on proper scoring rules, which are loss functions with the meaningful property that they incentivize the learner to predict ground-truth (conditional) probabilities. We assume two common representations of (epistemic) uncertainty, namely, in terms of a credal set, i.e. a set of probability distributions, or a second-order distribution, i.e., a distribution over probability distributions. Our framework establishes a natural bridge between these representations. We provide a formal justification of our approach and introduce new measures of epistemic and aleatoric uncertainty as concrete instantiations.

翻译：不确定性表示与量化在机器学习中至关重要，且是安全关键应用的重要前提。本文提出基于恰当评分规则的新型度量方法，用于量化偶然不确定性与认知不确定性。恰当评分规则作为一类损失函数，其关键特性在于能够激励学习器预测真实（条件）概率。我们假设两种常见的（认知）不确定性表示形式：置信集（即概率分布集合）与二阶分布（即概率分布上的分布）。该框架在这两种表示形式之间建立了天然的桥梁。我们为所提方法提供了形式化的理论依据，并引入新的认知不确定性与偶然不确定性度量作为具体实例化方案。