In the recent literature on machine learning and decision making, calibration has emerged as a desirable and widely-studied statistical property of the outputs of binary prediction models. However, the algorithmic aspects of measuring model calibration have remained relatively less well-explored. Motivated by [BGHN23], which proposed a rigorous framework for measuring distances to calibration, we initiate the algorithmic study of calibration through the lens of property testing. We define the problem of calibration testing from samples where given $n$ draws from a distribution $\mathcal{D}$ on $(predictions, binary outcomes)$, our goal is to distinguish between the case where $\mathcal{D}$ is perfectly calibrated, and the case where $\mathcal{D}$ is $\varepsilon$-far from calibration. We make the simple observation that the empirical smooth calibration linear program can be reformulated as an instance of minimum-cost flow on a highly-structured graph, and design an exact dynamic programming-based solver for it which runs in time $O(n\log^2(n))$, and solves the calibration testing problem information-theoretically optimally in the same time. This improves upon state-of-the-art black-box linear program solvers requiring $\Omega(n^\omega)$ time, where $\omega > 2$ is the exponent of matrix multiplication. We also develop algorithms for tolerant variants of our testing problem improving upon black-box linear program solvers, and give sample complexity lower bounds for alternative calibration measures to the one considered in this work. Finally, we present experiments showing the testing problem we define faithfully captures standard notions of calibration, and that our algorithms scale efficiently to accommodate large sample sizes.

翻译：在近期关于机器学习与决策制定的文献中，校准已成为二元预测模型输出中一种备受关注且被广泛研究的统计特性。然而，衡量模型校准性的算法层面问题仍相对缺乏深入探索。受[BGHN23]提出的测量校准距离的严格框架启发，我们首次从性质测试的角度出发，对校准性进行算法研究。我们定义了基于样本的校准测试问题：给定从（预测值，二元结果）上的分布$\mathcal{D}$中抽取的$n$个样本，目标是区分$\mathcal{D}$完全校准的情况与$\mathcal{D}$距离校准$\varepsilon$-远的情况。我们提出了一个简单观察：经验平滑校准线性规划可被重新表述为在一个高度结构化图上的最小成本流问题，并设计了一种基于动态规划的精确求解器，其运行时间为$O(n\log^2(n))，且在同一时间内信息论最优地解决了校准测试问题。这改进了当前需要$\Omega(n^\omega)$时间（其中$\omega > 2$是矩阵乘法指数）的黑盒线性规划求解器。我们还为测试问题的容忍变体开发了算法，其性能优于黑盒线性规划求解器，并针对本文所考虑校准度量之外的其他校准度量给出了样本复杂度下界。最后，我们通过实验表明，所定义的测试问题能够准确捕捉校准的标准概念，并且我们的算法能够高效扩展以适应大样本量。