Unsupervised Domain Adaptation (UDA) aims at classifying unlabeled target images leveraging source labeled ones. In this work, we consider the Partial Domain Adaptation (PDA) variant, where we have extra source classes not present in the target domain. Most successful algorithms use model selection strategies that rely on target labels to find the best hyper-parameters and/or models along training. However, these strategies violate the main assumption in PDA: only unlabeled target domain samples are available. Moreover, there are also inconsistencies in the experimental settings - architecture, hyper-parameter tuning, number of runs - yielding unfair comparisons. The main goal of this work is to provide a realistic evaluation of PDA methods with the different model selection strategies under a consistent evaluation protocol. We evaluate 7 representative PDA algorithms on 2 different real-world datasets using 7 different model selection strategies. Our two main findings are: (i) without target labels for model selection, the accuracy of the methods decreases up to 30 percentage points; (ii) only one method and model selection pair performs well on both datasets. Experiments were performed with our PyTorch framework, BenchmarkPDA, which we open source.

翻译：无人监督的域域适应(UDA) 旨在利用源标签的源代码对未贴标签的目标图像进行分类。 在这项工作中,我们考虑了部分域适应(PDA)变方,在目标域中我们没有额外的源类别。大多数成功的算法使用模型选择战略,依靠目标标签找到最佳超参数和/或培训中的最佳模型。然而,这些战略违反了PDA的主要假设:只有未贴标签的目标域样本才能得到。此外,在实验设置中也存在不一致之处 - 结构、超参数调、运行次数- 产生不公平的比较。这项工作的主要目标是根据一致的评价协议,对不同模式选择战略的PDA方法进行现实的评估。我们用7个不同的模型选择战略对2个不同的真实世界数据集的7个具有代表性的PDA算法进行了评估。我们的两个主要结论是:(一) 没有模型选择的目标标签,方法的准确性下降至30个百分点;(二) 只有一个方法和模型选择配方在两个数据集上表现良好。我们使用开放的 PyTorch 基准框架进行了实验。