A common explanation for the failure of out-of-distribution (OOD) generalization is that the model trained with empirical risk minimization (ERM) learns spurious features instead of the desired invariant features. However, several recent studies challenged this explanation and found that deep networks may have already learned sufficiently good features for OOD generalization. The debate extends to the in-distribution and OOD performance correlations along with training or fine-tuning neural nets across a variety of OOD generalization tasks. To understand these seemingly contradicting phenomena, we conduct a theoretical investigation and find that ERM essentially learns both spurious features and invariant features. On the other hand, the quality of learned features during ERM pre-training significantly affects the final OOD performance, as OOD objectives rarely learn new features. Failing to capture all the underlying useful features during pre-training will further limit the final OOD performance. To remedy the issue, we propose Feature Augmented Training (FAT ), to enforce the model to learn all useful features by retaining the already learned features and augmenting new ones by multiple rounds. In each round, the retention and augmentation operations are performed on different subsets of the training data that capture distinct features. Extensive experiments show that FAT effectively learns richer features and consistently improves the OOD performance when applied to various objectives.

翻译：关于分布外（OOD）泛化失败的一个常见解释是，采用经验风险最小化（ERM）训练的模型学习的是虚假特征而非期望的不变特征。然而，近期多项研究对这一解释提出质疑，发现深度网络可能已为OOD泛化学习了足够优质的特征。这一争议延伸至训练或微调神经网络过程中，分布内与OOD性能之间的相关性，且广泛存在于各类OOD泛化任务中。为理解这些看似矛盾的现象，我们开展理论研究，发现ERM实质上同时学习了虚假特征与不变特征。另一方面，ERM预训练阶段所学特征的质量显著影响最终OOD性能，这是因为OOD目标极少能学习新特征。若预训练阶段未能捕获所有潜在的有用特征，将进一步限制最终OOD性能。为解决该问题，我们提出特征增强训练（FAT），通过保留已学特征并分多轮增强新特征，强制模型学习全部有用特征。每一轮中，保留与增强操作均在捕获不同特征的训练数据子集上执行。大量实验表明，FAT能够有效学习更丰富的特征，且在应用于多种目标时持续提升OOD性能。