Average calibration of the prediction uncertainties of machine learning regression tasks can be tested in two ways: one is to estimate the calibration error (CE) as the difference between the mean absolute error (MSE) and the mean variance (MV) or mean squared uncertainty; the alternative is to compare the mean squared z-scores (ZMS) or scaled errors to 1. The problem is that both approaches might lead to different conclusions, as illustrated in this study for an ensemble of datasets from the recent machine learning uncertainty quantification (ML-UQ) literature. It is shown that the estimation of MV, MSE and their confidence intervals can become unreliable for heavy-tailed uncertainty and error distributions, which seems to be a common issue for ML-UQ datasets. By contrast, the ZMS statistic is less sensitive and offers the most reliable approach in this context. Unfortunately, the same problem affects also conditional calibrations statistics, such as the popular ENCE, and very likely post-hoc calibration methods based on similar statistics. As not much can be done to relieve this issue, except for a change of paradigm to intervals- or distribution-based UQ metrics, robust tailedness metrics are proposed to detect the potentially problematic datasets.

翻译：机器学习回归任务预测不确定性的平均校准可通过两种方式检验：一是估计校准误差（CE）作为平均绝对误差（MSE）与平均方差（MV）或平均平方不确定性之差；另一种方法是将平均平方z分数（ZMS）或缩放误差与1进行比较。问题在于，这两种方法可能得出不同结论——本研究基于近期机器学习不确定性量化（ML-UQ）文献中的数据集集合对此进行了验证。研究表明：对于重尾不确定性和误差分布，MV、MSE及其置信区间的估计可能变得不可靠，这似乎是ML-UQ数据集的常见问题。相比之下，ZMS统计量敏感性较低，在此类场景下提供了最可靠的检验手段。遗憾的是，同样的问题也影响条件校准统计量（如流行的ENCE），且极可能影响基于类似统计量的后验校准方法。由于除转向基于区间或分布的不确定性量化评估范式外，难以有效缓解该问题，本文提出鲁棒性峰度度量指标以检测潜在问题数据集。