Conformal prediction is a popular framework of uncertainty quantification that constructs prediction sets with coverage guarantees. To uphold the exchangeability assumption, many conformal prediction methods necessitate an additional holdout set for parameter tuning. Yet, the impact of violating this principle on coverage remains underexplored, making it ambiguous in practical applications. In this work, we empirically find that the tuning bias - the coverage gap introduced by leveraging the same dataset for tuning and calibration, is negligible for simple parameter tuning in many conformal prediction methods. In particular, we observe the scaling law of the tuning bias: this bias increases with parameter space complexity and decreases with calibration set size. Formally, we establish a theoretical framework to quantify the tuning bias and provide rigorous proof for the scaling law of the tuning bias by deriving its upper bound. In the end, we discuss how to reduce the tuning bias, guided by the theories we developed.

翻译：共形预测是一种流行的不确定性量化框架，能够构建具有覆盖保证的预测集。为满足可交换性假设，许多共形预测方法需要额外的留出集进行参数调整。然而，违反该原则对覆盖性能的影响尚未得到充分探索，导致实际应用中存在模糊性。本工作通过实证研究发现，对于许多共形预测方法中的简单参数调整，调参偏差——即利用相同数据集进行参数调整与校准所引入的覆盖差距——可以忽略不计。特别地，我们观察到调参偏差的缩放规律：该偏差随参数空间复杂度增加而增大，随校准集规模扩大而减小。形式上，我们建立了量化调参偏差的理论框架，并通过推导其上限为调参偏差的缩放规律提供了严格证明。最后，我们依据所发展的理论框架，探讨了如何有效降低调参偏差。