Uncertainty quantification is crucial for assessing the predictive ability of AI algorithms. Much research has been devoted to describing the predictive distribution (PD) $F(y|\mathbf{x})$ of a target variable $y \in \mathbb{R}$ given complex input features $\mathbf{x} \in \mathcal{X}$. However, off-the-shelf PDs (from, e.g., normalizing flows and Bayesian neural networks) often lack conditional calibration with the probability of occurrence of an event given input $\mathbf{x}$ being significantly different from the predicted probability. Current calibration methods do not fully assess and enforce conditionally calibrated PDs. Here we propose \texttt{Cal-PIT}, a method that addresses both PD diagnostics and recalibration by learning a single probability-probability map from calibration data. The key idea is to regress probability integral transform scores against $\mathbf{x}$. The estimated regression provides interpretable diagnostics of conditional coverage across the feature space. The same regression function morphs the misspecified PD to a re-calibrated PD for all $\mathbf{x}$. We benchmark our corrected prediction bands (a by-product of corrected PDs) against oracle bands and state-of-the-art predictive inference algorithms for synthetic data. We also provide results for two applications: (i) probabilistic nowcasting given sequences of satellite images, and (ii) conditional density estimation of galaxy distances given imaging data (so-called photometric redshift estimation). Our code is available as a Python package https://github.com/lee-group-cmu/Cal-PIT .

翻译：不确定性量化对于评估人工智能算法的预测能力至关重要。大量研究致力于描述给定复杂输入特征$\mathbf{x} \in \mathcal{X}$时目标变量$y \in \mathbb{R}$的预测分布（PD）$F(y|\mathbf{x})$。然而，现成预测分布（例如来自归一化流和贝叶斯神经网络）通常缺乏条件校准，即给定输入$\mathbf{x}$时事件发生概率与预测概率显著不一致。现有校准方法无法充分评估和实现条件校准的预测分布。本文提出\texttt{Cal-PIT}方法，通过从校准数据中学习单一概率-概率图来同时解决预测分布诊断和重校准问题。其核心思想是针对$\mathbf{x}$对概率积分变换得分进行回归。估计得到的回归提供了特征空间中条件覆盖率的可解释诊断。同一回归函数可将错误指定的预测分布转换为所有$\mathbf{x}$的重校准预测分布。我们将修正后的预测区间（修正预测分布的副产品）与合成数据上的基准区间及最先进的预测推理算法进行对比。此外，我们提供了两个应用案例：（i）基于卫星图像序列的概率临近预报；（ii）基于成像数据（即所谓的光度红移估计）的星系距离条件密度估计。我们的代码以Python包形式开源：https://github.com/lee-group-cmu/Cal-PIT。