The prediction accuracy of machine learning methods is steadily increasing, but the calibration of their uncertainty predictions poses a significant challenge. Numerous works focus on obtaining well-calibrated predictive models, but less is known about reliably assessing model calibration. This limits our ability to know when algorithms for improving calibration have a real effect, and when their improvements are merely artifacts due to random noise in finite datasets. In this work, we consider detecting mis-calibration of predictive models using a finite validation dataset as a hypothesis testing problem. The null hypothesis is that the predictive model is calibrated, while the alternative hypothesis is that the deviation from calibration is sufficiently large. We find that detecting mis-calibration is only possible when the conditional probabilities of the classes are sufficiently smooth functions of the predictions. When the conditional class probabilities are H\"older continuous, we propose T-Cal, a minimax optimal test for calibration based on a debiased plug-in estimator of the $\ell_2$-Expected Calibration Error (ECE). We further propose Adaptive T-Cal, a version that is adaptive to unknown smoothness. We verify our theoretical findings with a broad range of experiments, including with several popular deep neural net architectures and several standard post-hoc calibration methods. T-Cal is a practical general-purpose tool, which -- combined with classical tests for discrete-valued predictors -- can be used to test the calibration of virtually any probabilistic classification method.

翻译：机器学习方法的预测精度稳步提升，但其不确定性预测的校准仍是一个重大挑战。大量研究致力于获取良好校准的预测模型，但对于如何可靠评估模型校准效果却知之甚少。这限制了我们判断校准改进算法是否真正有效的能力，以及其改进是否仅为有限数据集随机噪声导致的伪影。本文将基于有限验证数据集检测预测模型的校准偏差视为假设检验问题：原假设为预测模型已校准，备择假设为校准偏差足够大。研究发现，仅当类别条件概率是预测值的充分光滑函数时，才能检测到校准偏差。在条件类别概率满足赫尔德连续性的条件下，我们提出T-Cal——一种基于$\ell_2$期望校准误差（ECE）去偏插件估计量的极小极大最优校准检验。进一步提出自适应T-Cal，该版本能自适应未知平滑度。我们通过涵盖多种深度神经网络架构与标准事后校准方法的广泛实验验证了理论发现。T-Cal是一种实用的通用工具，结合离散值预测器的经典检验方法，可用于检验几乎所有概率分类方法的校准性能。