We investigate a "learning to reject" framework to address the problem of silent failures in Domain Generalization (DG), where the test distribution differs from the training distribution. Assuming a mild distribution shift, we wish to accept out-of-distribution (OOD) data from a new domain whenever a model's estimated competence foresees trustworthy responses, instead of rejecting OOD data outright. Trustworthiness is then predicted via a proxy incompetence score that is tightly linked to the performance of a classifier. We present a comprehensive experimental evaluation of existing proxy scores as incompetence scores for classification and highlight the resulting trade-offs between rejection rate and accuracy gain. For comparability with prior work, we focus on standard DG benchmarks and consider the effect of measuring incompetence via different learned representations in a closed versus an open world setting. Our results suggest that increasing incompetence scores are indeed predictive of reduced accuracy, leading to significant improvements of the average accuracy below a suitable incompetence threshold. However, the scores are not yet good enough to allow for a favorable accuracy/rejection trade-off in all tested domains. Surprisingly, our results also indicate that classifiers optimized for DG robustness do not outperform a naive Empirical Risk Minimization (ERM) baseline in the competence region, that is, where test samples elicit low incompetence scores.

翻译：我们研究了一种“拒绝学习”框架，以解决领域泛化（Domain Generalization, DG）中测试分布与训练分布不同时的静默失败问题。在假设分布偏移温和的情况下，我们希望在模型估计的胜任能力预示可信响应时接受来自新领域的分布外（Out-of-Distribution, OOD）数据，而非完全拒绝OOD数据。进而，通过一个与分类器性能紧密相关的代理不胜任分数来预测可信度。我们对现有分类代理不胜任分数进行了全面的实验评估，并强调了拒绝率与准确率提升之间的权衡。为与先前工作保持可比性，我们聚焦于标准DG基准，并在封闭世界与开放世界设定下，考察通过不同学习表示衡量不胜任程度的效果。结果表明，增加不胜任分数确实能预测准确率下降，从而在合适的不胜任阈值以下显著提升平均准确率。然而，这些分数尚不足以在所有测试领域中实现有利的准确率/拒绝率权衡。令人意外的是，我们的结果还表明，针对DG鲁棒性优化的分类器在胜任区域（即测试样本引发低不胜任分数的区域）内并未优于朴素的经验风险最小化（Empirical Risk Minimization, ERM）基线。