Safe deployment of AI models requires proactive detection of potential prediction failures to prevent costly errors. While failure detection in classification problems has received significant attention, characterizing failure modes in regression tasks is more complicated and less explored. Existing approaches rely on epistemic uncertainties or feature inconsistency with the training distribution to characterize model risk. However, we show that uncertainties are necessary but insufficient to accurately characterize failure, owing to the various sources of error. In this paper, we propose PAGER (Principled Analysis of Generalization Errors in Regressors), a framework to systematically detect and characterize failures in deep regression models. Built upon the recently proposed idea of anchoring in deep models, PAGER unifies both epistemic uncertainties and novel, complementary non-conformity scores to organize samples into different risk regimes, thereby providing a comprehensive analysis of model errors. Additionally, we introduce novel metrics for evaluating failure detectors in regression tasks. We demonstrate the effectiveness of PAGER on synthetic and real-world benchmarks. Our results highlight the capability of PAGER to identify regions of accurate generalization and detect failure cases in out-of-distribution and out-of-support scenarios.

翻译：人工智能模型的安全部署需要主动检测潜在的预测失败，以防止代价高昂的错误。虽然分类问题中的失败检测已受到广泛关注，但回归任务中失效模式的表征更为复杂且研究较少。现有方法依赖认知不确定性或与训练分布的特征不一致性来表征模型风险。然而，我们证明不确定性虽必要但不足以准确表征失败，这是由于错误来源的多样性所致。本文提出PAGER（回归器泛化误差原则性分析）框架，用于系统性地检测和表征深度回归模型的失效。基于最近提出的深度模型锚定思想，PAGER统一了认知不确定性与新颖的互补非一致性分数，将样本组织到不同风险区间，从而提供对模型错误的全面分析。此外，我们引入了用于评估回归任务中失败检测器的新颖指标。在合成与真实世界基准上的实验验证了PAGER的有效性。结果表明，PAGER能够识别准确泛化区域，并检测出分布外与支持域外场景中的失败案例。