Endeavors have been recently made to leverage the vision transformer (ViT) for the challenging unsupervised domain adaptation (UDA) task. They typically adopt the cross-attention in ViT for direct domain alignment. However, as the performance of cross-attention highly relies on the quality of pseudo labels for targeted samples, it becomes less effective when the domain gap becomes large. We solve this problem from a game theory's perspective with the proposed model dubbed as PMTrans, which bridges source and target domains with an intermediate domain. Specifically, we propose a novel ViT-based module called PatchMix that effectively builds up the intermediate domain, i.e., probability distribution, by learning to sample patches from both domains based on the game-theoretical models. This way, it learns to mix the patches from the source and target domains to maximize the cross entropy (CE), while exploiting two semi-supervised mixup losses in the feature and label spaces to minimize it. As such, we interpret the process of UDA as a min-max CE game with three players, including the feature extractor, classifier, and PatchMix, to find the Nash Equilibria. Moreover, we leverage attention maps from ViT to re-weight the label of each patch by its importance, making it possible to obtain more domain-discriminative feature representations. We conduct extensive experiments on four benchmark datasets, and the results show that PMTrans significantly surpasses the ViT-based and CNN-based SoTA methods by +3.6% on Office-Home, +1.4% on Office-31, and +17.7% on DomainNet, respectively.

翻译：近年来，研究者开始尝试利用视觉变换器（ViT）解决具有挑战性的无监督域适应（UDA）任务。现有方法通常采用ViT中的交叉注意力机制直接进行域对齐。然而，由于交叉注意力的性能高度依赖于目标样本的伪标签质量，当域间隙增大时，其有效性会显著下降。我们从博弈论视角解决该问题，提出名为PMTrans的模型，通过构建中间域来连接源域与目标域。具体而言，我们设计了一种基于ViT的新型模块PatchMix，该模块基于博弈论模型学习从两个域中采样补丁，从而有效构建中间域（即概率分布）。通过这种方式，PatchMix学习混合源域与目标域的补丁以最大化交叉熵（CE），同时在特征空间与标签空间利用两种半监督混合损失来最小化交叉熵。因此，我们将UDA过程解释为由特征提取器、分类器和PatchMix三方参与的最小-最大交叉熵博弈，最终寻求纳什均衡解。此外，我们利用ViT的注意力图对每个补丁进行重要性加权，从而获得更具域判别性的特征表示。在四个基准数据集上的大量实验表明，PMTrans在Office-Home、Office-31和DomainNet上分别以+3.6%、+1.4%和+17.7%的显著优势超越基于ViT和CNN的现有最优方法。