Conformal prediction provides rigorous distribution-free finite-sample guarantees for marginal coverage under the assumption of exchangeability, but may exhibit systematic undercoverage or overcoverage for specific subpopulations. Assessing conditional validity is challenging, as standard stratification methods suffer from the curse of dimensionality. We propose Conformal Prediction Assessment (CPA), a framework that reframes the evaluation of conditional coverage as a supervised learning task by training a reliability estimator that predicts instance-level coverage probabilities. Building on this estimator, we introduce the Conditional Validity Index (CVI), which decomposes reliability into safety (undercoverage risk) and efficiency (overcoverage cost). We establish convergence rates for the reliability estimator and prove the consistency of CVI-based model selection. Extensive experiments on synthetic and real-world datasets demonstrate that CPA effectively diagnoses local failure modes and that CC-Select, our CVI-based model selection algorithm, consistently identifies predictors with superior conditional coverage performance.

翻译：共形预测在可交换性假设下提供了严格的分布自由有限样本边际覆盖保证，但可能对特定子群体表现出系统性欠覆盖或过覆盖。评估条件覆盖的合理性具有挑战性，因为标准分层方法存在维度灾难问题。我们提出共形预测评估（CPA）框架，通过训练可靠性估计器来预测实例级覆盖概率，将条件覆盖评估重新定义为监督学习任务。基于该估计器，我们引入条件有效性指数（CVI），将可靠性分解为安全性（欠覆盖风险）和效率（过覆盖成本）。我们建立了可靠性估计器的收敛速率，并证明了基于CVI的模型选择的一致性。在合成数据集和真实数据集上的大量实验表明，CPA能有效诊断局部失效模式，而我们的基于CVI的模型选择算法CC-Select能持续识别具有更优条件覆盖性能的预测器。