We investigate a "learning to reject" framework to address the problem of silent failures in Domain Generalization (DG), where the test distribution differs from the training distribution. Assuming a mild distribution shift, we wish to accept out-of-distribution (OOD) data from a new domain whenever a model's estimated competence foresees trustworthy responses, instead of rejecting OOD data outright. Trustworthiness is then predicted via a proxy incompetence score that is tightly linked to the performance of a classifier. We present a comprehensive experimental evaluation of existing proxy scores as incompetence scores for classification and highlight the resulting trade-offs between rejection rate and accuracy gain. For comparability with prior work, we focus on standard DG benchmarks and consider the effect of measuring incompetence via different learned representations in a closed versus an open world setting. Our results suggest that increasing incompetence scores are indeed predictive of reduced accuracy, leading to significant improvements of the average accuracy below a suitable incompetence threshold. However, the scores are not yet good enough to allow for a favorable accuracy/rejection trade-off in all tested domains. Surprisingly, our results also indicate that classifiers optimized for DG robustness do not outperform a naive Empirical Risk Minimization (ERM) baseline in the competence region, that is, where test samples elicit low incompetence scores.

翻译：我们研究了一种“学习拒绝”框架，以解决域泛化（DG）中测试分布与训练分布不同时的静默失败问题。在温和分布偏移假设下，我们希望当模型估计的能力预见到可信响应时，接受来自新域的分布外（OOD）数据，而非直接拒绝所有OOD数据。可信度通过一个与分类器性能紧密相关的代理不胜任分数来预测。我们对现有代理分数作为分类不胜任分数进行了全面的实验评估，并揭示了拒绝率与准确率提升之间的权衡。为与先前工作可比，我们聚焦于标准DG基准，并考虑在封闭与开放世界设置中通过不同学习表示测量不胜任的影响。结果表明，增加的不胜任分数确实能预测准确率下降，从而在适当的不胜任阈值下显著提升平均准确率。然而，这些分数尚不足以在所有测试域中实现有利的准确率/拒绝权衡。令人惊讶的是，我们的结果还表明，针对DG鲁棒性优化的分类器在能力区域（即测试样本引发低不胜任分数的区域）并未优于朴素的经验风险最小化（ERM）基线。