Reliably estimating the uncertainty of a prediction throughout the model lifecycle is crucial in many safety-critical applications. The most common way to measure this uncertainty is via the predicted confidence. While this tends to work well for in-domain samples, these estimates are unreliable under domain drift and restricted to classification. Alternatively, proper scores can be used for most predictive tasks but a bias-variance decomposition for model uncertainty does not exist in the current literature. In this work we introduce a general bias-variance decomposition for proper scores, giving rise to the Bregman Information as the variance term. We discover how exponential families and the classification log-likelihood are special cases and provide novel formulations. Surprisingly, we can express the classification case purely in the logit space. We showcase the practical relevance of this decomposition on several downstream tasks, including model ensembles and confidence regions. Further, we demonstrate how different approximations of the instance-level Bregman Information allow reliable out-of-distribution detection for all degrees of domain drift.

翻译：在诸多安全关键应用中，可靠地估计模型整个生命周期内预测的不确定性至关重要。衡量这种不确定性最常用的方式是通过预测置信度。虽然这种方法通常对领域内样本有效，但在领域漂移下这些估计并不可靠，且仅限于分类任务。另一种方法是，对大多数预测任务可采用恰当评分函数，但当前文献中尚不存在模型不确定性的偏差-方差分解方法。本研究引入了一种针对恰当评分函数的广义偏差-方差分解，从而将布雷格曼信息确定为方差项。我们发现指数族分布和分类对数似然是其特例，并提出了新颖的公式表达。令人惊讶的是，我们能够在对数几率空间中纯粹地表达分类情形。我们通过模型集成和置信区域等多项下游任务展示了该分解的实际相关性。此外，我们证明了实例级布雷格曼信息的不同近似方法能够针对所有程度的领域漂移实现可靠的分布外检测。