The prediction accuracy of machine learning methods is steadily increasing, but the calibration of their uncertainty predictions poses a significant challenge. Numerous works focus on obtaining well-calibrated predictive models, but less is known about reliably assessing model calibration. This limits our ability to know when algorithms for improving calibration have a real effect, and when their improvements are merely artifacts due to random noise in finite datasets. In this work, we consider detecting mis-calibration of predictive models using a finite validation dataset as a hypothesis testing problem. The null hypothesis is that the predictive model is calibrated, while the alternative hypothesis is that the deviation from calibration is sufficiently large. We find that detecting mis-calibration is only possible when the conditional probabilities of the classes are sufficiently smooth functions of the predictions. When the conditional class probabilities are H\"older continuous, we propose T-Cal, a minimax optimal test for calibration based on a debiased plug-in estimator of the $\ell_2$-Expected Calibration Error (ECE). We further propose Adaptive T-Cal, a version that is adaptive to unknown smoothness. We verify our theoretical findings with a broad range of experiments, including with several popular deep neural net architectures and several standard post-hoc calibration methods. T-Cal is a practical general-purpose tool, which -- combined with classical tests for discrete-valued predictors -- can be used to test the calibration of virtually any probabilistic classification method.

翻译：机器学习方法的预测精度正稳步提升，但其不确定性预测的校准问题仍构成重大挑战。大量研究致力于获得良好校准的预测模型，但对可靠评估模型校准的认知却相对有限。这限制了我们判断校准改进算法是否真正有效的能力，以及判断其改进是否仅是有限数据集随机噪声导致的伪像。本文将基于有限验证数据集检测预测模型误校准问题视为假设检验课题。原假设为预测模型已校准，备择假设则为校准偏差显著较大。研究发现，仅当各类别条件概率作为预测值的函数具有充分光滑性时，误校准检测才具有可行性。当条件类别概率满足赫尔德连续条件时，我们提出T-Cal——一种基于去偏插入估计器的最小最大最优校准检验方法，该估计器针对$\ell_2$期望校准误差（ECE）设计。进一步提出自适应T-Cal，该方法能自适应未知光滑性。我们通过涵盖多种主流深度神经网络架构与标准事后校准方法的广泛实验验证了理论发现。T-Cal作为实用通用工具，可与离散值预测器的经典检验结合，适用于几乎所有概率分类方法的校准检验。