Effective machine learning models learn both robust features that directly determine the outcome of interest (e.g., an object with wheels is more likely to be a car), and shortcut features (e.g., an object on a road is more likely to be a car). The latter can be a source of error under distributional shift, when the correlations change at test-time. The prevailing sentiment in the robustness literature is to avoid such correlative shortcut features and learn robust predictors. However, while robust predictors perform better on worst-case distributional shifts, they often sacrifice accuracy on majority subpopulations. In this paper, we argue that shortcut features should not be entirely discarded. Instead, if we can identify the subpopulation to which an input belongs, we can adaptively choose among models with different strengths to achieve high performance on both majority and minority subpopulations. We propose COnfidence-baSed MOdel Selection (CosMoS), where we observe that model confidence can effectively guide model selection. Notably, CosMoS does not require any target labels or group annotations, either of which may be difficult to obtain or unavailable. We evaluate CosMoS on four datasets with spurious correlations, each with multiple test sets with varying levels of data distribution shift. We find that CosMoS achieves 2-5% lower average regret across all subpopulations, compared to using only robust predictors or other model aggregation methods.

翻译：有效的机器学习模型既能学习直接决定目标结果的鲁棒特征（例如，有轮子的物体更可能是汽车），也能学习捷径特征（例如，位于道路上的物体更可能是汽车）。当测试时相关性发生变化，后者在分布偏移下可能成为误差来源。鲁棒性文献中的主流观点是避免此类关联性捷径特征并学习鲁棒预测器。然而，尽管鲁棒预测器在最坏情况下的分布偏移中表现更优，它们常会牺牲多数子群体的准确性。本文认为，捷径特征不应被完全摒弃。相反，若我们能识别输入所属的子群体，即可自适应地选择具有不同优势的模型，从而在多数和少数子群体上均实现高性能。我们提出基于置信度的模型选择方法（CosMoS），观察到模型置信度可有效指导模型选择。值得注意的是，CosMoS无需任何目标标签或群体标注，而这两者可能难以获取或无法获得。我们在四个含虚假相关的数据集上评估CosMoS，每个数据集包含多个具有不同程度数据分布偏移的测试集。研究发现，与仅使用鲁棒预测器或其他模型聚合方法相比，CosMoS在所有子群体上的平均遗憾值降低了2-5%。