We consider the problem of uncertainty quantification for a pretrained classification model deployed under unknown distribution shift. We propose Audited Conformal Prediction (ACP), a method that leverages a small labeled dataset from the target population to train an auxiliary audit model identifying inputs where the legacy model is likely to fail. By integrating the audit model's outputs into the conformal prediction framework, ACP produces prediction sets that guarantee marginal coverage while achieving substantially higher conditional coverage in practice than existing approaches. We develop and analyze two complementary integration strategies -- one targeting marginal coverage with improved conditional performance, the other providing explicit group-conditional coverage guarantees -- and establish theoretical guarantees for both. Experiments on synthetic and real-world datasets validate the method and illustrate trade-offs between prediction set size and conditional coverage.

翻译：我们研究了预训练分类模型在未知分布漂移场景下部署时的不确定性量化问题。本文提出审核性共形预测方法，通过利用目标群体的小规模标注数据集训练辅助审核模型，识别原始模型可能失效的输入样本。将该审核模型的输出整合至共形预测框架后，所提方法生成的预测集既能保证边际覆盖，在实际应用中又可获得显著优于现有方法的条件覆盖表现。我们开发并分析了两种互补性整合策略——一种面向边际覆盖以改善条件性能，另一种提供明确的群组条件覆盖保证——同时为两者建立了理论保障。合成数据集与真实世界数据集的实验验证了该方法的有效性，并揭示了预测集规模与条件覆盖之间的权衡关系。