Modern image classifiers are very accurate, but the predictions come without uncertainty estimates. Conformal predictors provide uncertainty estimates by computing a set of classes containing the correct class with a user-specified probability based on the classifier's probability estimates. To provide such sets, conformal predictors often estimate a cutoff threshold for the probability estimates based on a calibration set. Conformal predictors guarantee reliability only when the calibration set is from the same distribution as the test set. Therefore, conformal predictors need to be recalibrated for new distributions. However, in practice, labeled data from new distributions is rarely available, making calibration infeasible. In this work, we consider the problem of predicting the cutoff threshold for a new distribution based on unlabeled examples. While it is impossible in general to guarantee reliability when calibrating based on unlabeled examples, we propose a method that provides excellent uncertainty estimates under natural distribution shifts, and provably works for a specific model of a distribution shift.

翻译：现代图像分类器非常精确，但预测结果缺乏不确定性估计。共形预测器通过根据分类器的概率估计计算包含正确类别的类别集合（以用户指定的概率）来提供不确定性估计。为提供此类集合，共形预测器通常基于校准集对概率估计的截断阈值进行估计。共形预测器仅在校准集与测试集同分布时才能保证可靠性。因此，针对新分布需要重新校准共形预测器。然而实践中新分布的标记数据很少可用，导致校准不可行。本文研究基于无标签样本预测新分布截断阈值的问题。虽然通常无法保证基于无标签样本校准的可靠性，但我们提出了一种方法，可在自然分布偏移下提供优秀的不确定性估计，并能在特定分布偏移模型下证明其有效性。