Existing test-time adaptation (TTA) approaches often adapt models with the unlabeled testing data stream. A recent attempt relaxed the assumption by introducing limited human annotation, referred to as Human-In-the-Loop Test-Time Adaptation (HILTTA) in this study. The focus of existing HILTTA studies lies in selecting the most informative samples to label, a.k.a. active learning. In this work, we are motivated by a pitfall of TTA, i.e. sensitivity to hyper-parameters, and propose to approach HILTTA by synergizing active learning and model selection. Specifically, we first select samples for human annotation (active learning) and then use the labeled data to select optimal hyper-parameters (model selection). To prevent the model selection process from overfitting to local distributions, multiple regularization techniques are employed to complement the validation objective. A sample selection strategy is further tailored by considering the balance between active learning and model selection purposes. We demonstrate on 5 TTA datasets that the proposed HILTTA approach is compatible with off-the-shelf TTA methods and such combinations substantially outperform the state-of-the-art HILTTA methods. Importantly, our proposed method can always prevent choosing the worst hyper-parameters on all off-the-shelf TTA methods. The source code is available at https://github.com/Yushu-Li/HILTTA.

翻译：现有的测试时适应方法通常利用未标注的测试数据流来调整模型。近期研究通过引入有限的人工标注放宽了这一假设，本研究将其称为人机协同测试时适应。现有HILTTA研究的重点在于选择信息量最大的样本进行标注，即主动学习。本研究受TTA方法的一个缺陷——对超参数的敏感性——所启发，提出通过协同主动学习与模型选择来实现HILTTA。具体而言，我们首先选择样本进行人工标注，然后利用标注数据选择最优超参数。为防止模型选择过程对局部分布过拟合，我们采用多种正则化技术来补充验证目标。通过权衡主动学习与模型选择的目标，我们进一步定制了样本选择策略。在5个TTA数据集上的实验表明，所提出的HILTTA方法可与现成的TTA方法兼容，且这种组合显著优于当前最先进的HILTTA方法。重要的是，我们提出的方法在所有现成TTA方法上均能有效避免选择最差超参数。源代码已公开于https://github.com/Yushu-Li/HILTTA。