Human visual perception can easily generalize to out-of-distributed visual data, which is far beyond the capability of modern machine learning models. Domain generalization (DG) aims to close this gap, with existing DG methods mainly focusing on the loss function design. In this paper, we propose to explore an orthogonal direction, i.e., the design of the backbone architecture. It is motivated by an empirical finding that transformer-based models trained with empirical risk minimization (ERM) outperform CNN-based models employing state-of-the-art (SOTA) DG algorithms on multiple DG datasets. We develop a formal framework to characterize a network's robustness to distribution shifts by studying its architecture's alignment with the correlations in the dataset. This analysis guides us to propose a novel DG model built upon vision transformers, namely Generalizable Mixture-of-Experts (GMoE). Extensive experiments on DomainBed demonstrate that GMoE trained with ERM outperforms SOTA DG baselines by a large margin. Moreover, GMoE is complementary to existing DG methods and its performance is substantially improved when trained with DG algorithms.

翻译：人类视觉感知可以轻松泛化到分布外的视觉数据，这远超现代机器学习模型的能力。领域泛化旨在弥合这一差距，现有领域泛化方法主要关注损失函数的设计。本文提出探索一个正交方向，即主干架构的设计。这一方向的动机来自实证发现：基于Transformer的模型在使用经验风险最小化训练时，在多个领域泛化数据集上优于采用最先进领域泛化算法的CNN模型。我们建立了一个形式化框架，通过研究网络架构与数据集相关性的对齐程度，来刻画网络对分布偏移的鲁棒性。这一分析引导我们提出基于视觉Transformer的新型领域泛化模型——可泛化混合专家模型。在DomainBed上的大量实验表明，使用经验风险最小化训练的GMoE以较大优势超越最先进的领域泛化基线。此外，GMoE与现有领域泛化方法具有互补性，当结合领域泛化算法训练时，其性能会显著提升。