Despite superior performance in many situations, deep neural networks are often vulnerable to adversarial examples and distribution shifts, limiting model generalization ability in real-world applications. To alleviate these problems, recent approaches leverage distributional robustness optimization (DRO) to find the most challenging distribution, and then minimize loss function over this most challenging distribution. Regardless of achieving some improvements, these DRO approaches have some obvious limitations. First, they purely focus on local regularization to strengthen model robustness, missing a global regularization effect which is useful in many real-world applications (e.g., domain adaptation, domain generalization, and adversarial machine learning). Second, the loss functions in the existing DRO approaches operate in only the most challenging distribution, hence decouple with the original distribution, leading to a restrictive modeling capability. In this paper, we propose a novel regularization technique, following the veins of Wasserstein-based DRO framework. Specifically, we define a particular joint distribution and Wasserstein-based uncertainty, allowing us to couple the original and most challenging distributions for enhancing modeling capability and applying both local and global regularizations. Empirical studies on different learning problems demonstrate that our proposed approach significantly outperforms the existing regularization approaches in various domains: semi-supervised learning, domain adaptation, domain generalization, and adversarial machine learning.

翻译：尽管深度神经网络在许多场景下表现优异，但常易受到对抗样本和分布偏移的影响，这限制了模型在现实应用中的泛化能力。为解决这些问题，近期方法利用分布鲁棒性优化（DRO）寻找最具挑战性的分布，并在此分布上最小化损失函数。尽管取得了一定改进，这些DRO方法仍存在明显局限。首先，它们仅关注局部正则化以增强模型鲁棒性，缺失了在诸多现实应用（如域适应、域泛化及对抗机器学习）中具有重要价值的全局正则化效果。其次，现有DRO方法的损失函数仅在最具挑战性的分布上运行，由此与原分布解耦，导致建模能力受限。本文基于Wasserstein型DRO框架，提出一种新颖的正则化技术。具体而言，我们定义了一种特定的联合分布和基于Wasserstein的不确定性度量，从而能够耦合原始分布与最具挑战性的分布以增强建模能力，并同时施加局部与全局正则化。针对不同学习问题的实证研究表明，所提方法在半监督学习、域适应、域泛化及对抗机器学习等多个领域显著优于现有正则化方法。