Uncertainty quantification is crucial for assessing the predictive ability of AI algorithms. Much research has been devoted to describing the predictive distribution (PD) $F(y|\mathbf{x})$ of a target variable $y \in \mathbb{R}$ given complex input features $\mathbf{x} \in \mathcal{X}$. However, off-the-shelf PDs (from, e.g., normalizing flows and Bayesian neural networks) often lack conditional calibration with the probability of occurrence of an event given input $\mathbf{x}$ being significantly different from the predicted probability. Current calibration methods do not fully assess and enforce conditionally calibrated PDs. Here we propose \texttt{Cal-PIT}, a method that addresses both PD diagnostics and recalibration by learning a single probability-probability map from calibration data. The key idea is to regress probability integral transform scores against $\mathbf{x}$. The estimated regression provides interpretable diagnostics of conditional coverage across the feature space. The same regression function morphs the misspecified PD to a re-calibrated PD for all $\mathbf{x}$. We benchmark our corrected prediction bands (a by-product of corrected PDs) against oracle bands and state-of-the-art predictive inference algorithms for synthetic data. We also provide results for two applications: (i) probabilistic nowcasting given sequences of satellite images, and (ii) conditional density estimation of galaxy distances given imaging data (so-called photometric redshift estimation). Our code is available as a Python package https://github.com/lee-group-cmu/Cal-PIT .

翻译：不确定性量化对于评估AI算法的预测能力至关重要。大量研究致力于描述给定复杂输入特征$\mathbf{x} \in \mathcal{X}$时目标变量$y \in \mathbb{R}$的预测分布$F(y|\mathbf{x})$。然而，现成的预测分布（例如来自归一化流和贝叶斯神经网络的预测分布）通常缺乏条件校准，即给定输入$\mathbf{x}$时事件发生的概率与预测概率存在显著差异。当前的校准方法无法全面评估和强制执行条件校准的预测分布。本文提出\texttt{Cal-PIT}方法，通过从校准数据中学习单一概率-概率映射，同时解决预测分布诊断和重新校准问题。关键思想是回归概率积分变换分数对$\mathbf{x}$的依赖关系。估计的回归提供跨特征空间条件覆盖的可解释诊断。相同的回归函数将错误指定的预测分布转换为针对所有$\mathbf{x}$的重新校准预测分布。我们将修正后的预测区间（修正预测分布的副产品）与合成数据上的理想区间及最先进的预测推断算法进行基准测试。我们还提供了两个应用的结果：(i) 基于卫星图像序列的概率临近预报，以及(ii) 基于成像数据的星系距离条件密度估计（即所谓的光度红移估计）。我们的代码以Python包形式发布在 https://github.com/lee-group-cmu/Cal-PIT。