Recently, there has been a growing surge of interest in enabling machine learning systems to generalize well to Out-of-Distribution (OOD) data. Most efforts are devoted to advancing optimization objectives that regularize models to capture the underlying invariance; however, there often are compromises in the optimization process of these OOD objectives: i) Many OOD objectives have to be relaxed as penalty terms of Empirical Risk Minimization (ERM) for the ease of optimization, while the relaxed forms can weaken the robustness of the original objective; ii) The penalty terms also require careful tuning of the penalty weights due to the intrinsic conflicts between ERM and OOD objectives. Consequently, these compromises could easily lead to suboptimal performance of either the ERM or OOD objective. To address these issues, we introduce a multi-objective optimization (MOO) perspective to understand the OOD optimization process, and propose a new optimization scheme called PAreto Invariant Risk Minimization (PAIR). PAIR improves the robustness of OOD objectives by cooperatively optimizing with other OOD objectives, thereby bridging the gaps caused by the relaxations. Then PAIR approaches a Pareto optimal solution that trades off the ERM and OOD objectives properly. Extensive experiments on challenging benchmarks, WILDS, show that PAIR alleviates the compromises and yields top OOD performances.

翻译：近年来，机器学习系统在分布外数据上实现良好泛化的研究兴趣日益增长。大多数工作致力于改进优化目标，通过正则化手段约束模型以捕捉潜在的不变性；然而，在优化这些分布外目标时往往存在妥协：i) 许多分布外目标需简化为经验风险最小化的惩罚项以方便优化，但这种简化形式可能削弱原始目标的鲁棒性；ii) 由于经验风险最小化与分布外目标之间存在内在冲突，惩罚项还需要仔细调节惩罚权重。这些妥协容易导致经验风险最小化或分布外目标中的某一方性能欠佳。为解决这些问题，我们从多目标优化的视角理解分布外优化过程，并提出名为帕累托不变风险最小化的新优化方案。帕累托不变风险最小化通过与其他分布外目标协同优化来增强分布外目标的鲁棒性，从而弥合因简化造成的鸿沟。进而，该方法能够逼近一个合理权衡经验风险最小化与分布外目标的最优帕累托解。在具有挑战性的基准测试WILDS上的大量实验表明，帕累托不变风险最小化有效缓解了上述妥协问题，并取得了顶尖的分布外性能。