Unsupervised domain adaptation (UDA) methods facilitate the transfer of models to target domains without labels. However, these methods necessitate a labeled target validation set for hyper-parameter tuning and model selection. In this paper, we aim to find an evaluation metric capable of assessing the quality of a transferred model without access to target validation labels. We begin with the metric based on mutual information of the model prediction. Through empirical analysis, we identify three prevalent issues with this metric: 1) It does not account for the source structure. 2) It can be easily attacked. 3) It fails to detect negative transfer caused by the over-alignment of source and target features. To address the first two issues, we incorporate source accuracy into the metric and employ a new MLP classifier that is held out during training, significantly improving the result. To tackle the final issue, we integrate this enhanced metric with data augmentation, resulting in a novel unsupervised UDA metric called the Augmentation Consistency Metric (ACM). Additionally, we empirically demonstrate the shortcomings of previous experiment settings and conduct large-scale experiments to validate the effectiveness of our proposed metric. Furthermore, we employ our metric to automatically search for the optimal hyper-parameter set, achieving superior performance compared to manually tuned sets across four common benchmarks. Codes will be available soon.

翻译：无监督领域自适应（UDA）方法能够在无标签条件下将模型迁移至目标领域。然而，这些方法仍需标注的目标验证集进行超参数调优与模型选择。本文旨在探究一种无需目标验证标签即可评估迁移模型质量的评价指标。我们首先基于模型预测的互信息构建指标，通过实证分析发现该指标存在三个普遍问题：1）未考虑源域结构；2）易受攻击；3）无法检测因源域与目标域特征过度对齐导致的负迁移。针对前两个问题，我们将源域准确率融入指标，并采用训练阶段冻结的新型多层感知机分类器，显著提升了评估效果。为解决第三个问题，我们将改进后的指标与数据增强技术相结合，提出名为"增强一致性指标（ACM）"的新型无监督UDA评估指标。此外，我们通过实证揭示了先前实验设置的缺陷，并通过大规模实验验证了所提指标的有效性。进一步地，我们利用该指标自动搜索最优超参数组合，在四个通用基准上取得了优于人工调参的性能。代码即将开源。