Empirical risk minimization (ERM) is known in practice to be non-robust to distributional shift where the training and the test distributions are different. A suite of approaches, such as importance weighting, and variants of distributionally robust optimization (DRO), have been proposed to solve this problem. But a line of recent work has empirically shown that these approaches do not significantly improve over ERM in real applications with distribution shift. The goal of this work is to obtain a comprehensive theoretical understanding of this intriguing phenomenon. We first posit the class of Generalized Reweighting (GRW) algorithms, as a broad category of approaches that iteratively update model parameters based on iterative reweighting of the training samples. We show that when overparameterized models are trained under GRW, the resulting models are close to that obtained by ERM. We also show that adding small regularization which does not greatly affect the empirical training accuracy does not help. Together, our results show that a broad category of what we term GRW approaches are not able to achieve distributionally robust generalization. Our work thus has the following sobering takeaway: to make progress towards distributionally robust generalization, we either have to develop non-GRW approaches, or perhaps devise novel classification/regression loss functions that are adapted to the class of GRW approaches.

翻译：经验风险最小化（ERM）在实践中已知对训练分布与测试分布不同的分布偏移缺乏鲁棒性。为解决此问题，已有多种方法被提出，例如重要性加权以及分布稳健优化（DRO）的变体。但近期一系列工作通过实验表明，在存在分布偏移的实际应用中，这些方法并未显著优于ERM。本文旨在对这一引人注目的现象提供全面的理论理解。我们首先提出广义重加权（GRW）算法类别，这是一类基于训练样本迭代重加权来更新模型参数的广泛方法。我们证明，当使用GRW训练过参数化模型时，所得模型与ERM获得的模型接近。我们还表明，添加不影响经验训练准确率的小正则化也无济于事。综合来看，我们的结果表明，我们称之为GRW的这类广泛方法无法实现分布稳健泛化。因此，本文得出以下发人深省的结论：要推进分布稳健泛化，我们必须要么开发非GRW方法，要么设计适应GRW方法类别的新型分类/回归损失函数。