Conformal prediction(CP) has become a cornerstone of distribution-free uncertainty quantification, conventionally evaluated by its coverage and interval length. This work critically examines the sufficiency of these standard metrics. We demonstrate that the interval length might be deceptively improved through a counter-intuitive approach termed Prejudicial Trick(PT), while the coverage remains valid. Specifically, for any given test sample, PT probabilistically returns an interval, which is either null or constructed using an adjusted confidence level, thereby preserving marginal coverage. While PT potentially yields a deceptively lower interval length, it introduces practical vulnerabilities: the same input can yield completely different prediction intervals across repeated runs of the algorithm. We formally derive the conditions under which PT achieves these misleading improvements and provide extensive empirical evidence across various regression and classification tasks. Furthermore, we introduce a new metric interval stability which helps detect whether a new CP method implicitly improves the length based on such PT-like techniques. Code is available at https://github.com/benben-cd/PT-Conformal-Prediction.

翻译：共形预测（CP）已成为无分布假设不确定性量化的基石，传统上通过其覆盖率和区间长度进行评估。本研究批判性地审视了这些标准指标的充分性。我们证明，通过一种被称为“偏见技巧”（PT）的反直觉方法，区间长度可能被欺骗性地优化，同时保持覆盖率有效。具体而言，对于任意给定测试样本，PT以概率方式返回一个区间——该区间要么为空，要么基于调整后的置信水平构建——从而维持边际覆盖率。虽然PT可能产生具有欺骗性的较短区间长度，但它引入了实际漏洞：同一输入在算法多次运行中可能产生完全不同的预测区间。我们正式推导了PT实现这些误导性改进的条件，并在各种回归和分类任务中提供了广泛的经验证据。此外，我们引入了一个新指标——“区间稳定性”，用于帮助检测新的CP方法是否隐式地通过类似PT的技术优化区间长度。代码见https://github.com/benben-cd/PT-Conformal-Prediction。