The existence of spurious correlations such as image backgrounds in the training environment can make empirical risk minimization (ERM) perform badly in the test environment. To address this problem, Kirichenko et al. (2022) empirically found that the core features that are related to the outcome can still be learned well even with the presence of spurious correlations. This opens a promising strategy to first train a feature learner rather than a classifier, and then perform linear probing (last layer retraining) in the test environment. However, a theoretical understanding of when and why this approach works is lacking. In this paper, we find that core features are only learned well when their associated non-realizable noise is smaller than that of spurious features, which is not necessarily true in practice. We provide both theories and experiments to support this finding and to illustrate the importance of non-realizable noise. Moreover, we propose an algorithm called Freeze then Train (FTT), that first freezes certain salient features and then trains the rest of the features using ERM. We theoretically show that FTT preserves features that are more beneficial to test time probing. Across two commonly used spurious correlation datasets, FTT outperforms ERM, IRM, JTT and CVaR-DRO, with substantial improvement in accuracy (by 4.5%) when the feature noise is large. FTT also performs better on general distribution shift benchmarks.

翻译：训练环境中存在的虚假相关性（如图像背景）会导致经验风险最小化（ERM）在测试环境中表现不佳。为解决此问题，Kirichenko等人（2022）通过实验发现，即便存在虚假相关性，与结果相关的核心特征仍可被有效学习。这开辟了一种有前景的策略：首先训练特征学习器而非分类器，再在测试环境中进行线性探测（最后一层重新训练）。然而，目前尚缺乏对该方法何时有效及其背后机理的理论理解。本文发现，仅当核心特征相关的非可实现噪声小于虚假特征时，核心特征才能被充分学习——这在实践中并非必然成立。我们通过理论推导与实验验证支持该发现，并阐明了非可实现噪声的重要性。此外，我们提出一种名为"先冻结再训练"（FTT）的算法，该算法先冻结某些显著特征，然后使用ERM训练其余特征。理论证明，FTT能保留对测试阶段探测更有利的特征。在两个常用虚假相关性数据集上，当特征噪声较大时，FTT的准确率显著优于ERM、IRM、JTT和CVaR-DRO（提升4.5%）。FTT在通用分布偏移基准测试中也表现出更优性能。