A decision-theoretic characterization of perfect calibration is that an agent seeking to minimize a proper loss in expectation cannot improve their outcome by post-processing a perfectly calibrated predictor. Hu and Wu (FOCS'24) use this to define an approximate calibration measure called calibration decision loss ($\mathsf{CDL}$), which measures the maximal improvement achievable by any post-processing over any proper loss. Unfortunately, $\mathsf{CDL}$ turns out to be intractable to even weakly approximate in the offline setting, given black-box access to the predictions and labels. We suggest circumventing this by restricting attention to structured families of post-processing functions $K$. We define the calibration decision loss relative to $K$, denoted $\mathsf{CDL}_K$ where we consider all proper losses but restrict post-processings to a structured family $K$. We develop a comprehensive theory of when $\mathsf{CDL}_K$ is information-theoretically and computationally tractable, and use it to prove both upper and lower bounds for natural classes $K$. In addition to introducing new definitions and algorithmic techniques to the theory of calibration for decision making, our results give rigorous guarantees for some widely used recalibration procedures in machine learning.

翻译：完美校准的决策理论特征在于：若一个智能体寻求最小化适当损失的期望值，则无法通过后处理一个完美校准的预测器来改善其结果。Hu与Wu（FOCS'24）基于此定义了近似校准度量——校准决策损失（$\\mathsf{CDL}$），该度量衡量了在任何适当损失下，通过任意后处理所能实现的最大改进程度。然而，在离线设置中，即使仅需对$\\mathsf{CDL}$进行弱近似，在仅能黑盒访问预测结果与标签的条件下，该问题也被证明是难解的。我们建议通过将关注范围限制在结构化后处理函数族$K$中来规避此难题。我们定义了相对于$K$的校准决策损失，记为$\\mathsf{CDL}_K$，其中我们考虑所有适当损失，但将后处理限制在结构化族$K$内。我们建立了关于$\\mathsf{CDL}_K$在信息论与计算层面何时可处理的完整理论，并利用该理论对自然类别$K$证明了上界与下界。除了为决策校准理论引入新的定义与算法技术外，我们的研究结果为机器学习中一些广泛使用的再校准流程提供了严格的理论保证。