Since real-world machine systems are running in non-stationary environments, Continual Test-Time Adaptation (CTTA) task is proposed to adapt the pre-trained model to continually changing target domains. Recently, existing methods mainly focus on model-based adaptation, which aims to leverage a self-training manner to extract the target domain knowledge. However, pseudo labels can be noisy and the updated model parameters are unreliable under dynamic data distributions, leading to error accumulation and catastrophic forgetting in the continual adaptation process. To tackle these challenges and maintain the model plasticity, we design a Visual Domain Adapter (ViDA) for CTTA, explicitly handling both domain-specific and domain-shared knowledge. Specifically, we first comprehensively explore the different domain representations of the adapters with trainable high-rank or low-rank embedding spaces. Then we inject ViDAs into the pre-trained model, which leverages high-rank and low-rank features to adapt the current domain distribution and maintain the continual domain-shared knowledge, respectively. To exploit the low-rank and high-rank ViDAs more effectively, we further propose a Homeostatic Knowledge Allotment (HKA) strategy, which adaptively combines different knowledge from each ViDA. Extensive experiments conducted on four widely used benchmarks demonstrate that our proposed method achieves state-of-the-art performance in both classification and segmentation CTTA tasks. Note that, our method can be regarded as a novel transfer paradigm for large-scale models, delivering promising results in adaptation to continually changing distributions. Project page: https://sites.google.com/view/iclr2024-vida/home.

翻译：现实世界中的机器系统运行在非平稳环境下，因此提出持续测试时适应任务，旨在使预训练模型适应持续变化的目标域。现有方法主要聚焦于基于模型的适应，通过自训练方式提取目标域知识。然而，在动态数据分布下，伪标签可能存在噪声，且更新的模型参数不可靠，导致持续适应过程中出现误差累积与灾难性遗忘。为解决上述挑战并保持模型可塑性，我们为CTTA设计了视觉域适配器，显式处理域特定知识与域共享知识。具体而言，我们首先全面探索了具有可训练高秩或低秩嵌入空间的适配器所呈现的不同域表示；随后将ViDA注入预训练模型，分别利用高秩与低秩特征适配当前域分布并维持持续域共享知识。为更有效利用低秩与高秩ViDA，我们进一步提出稳态知识分配策略，自适应整合各ViDA的不同知识。在四个广泛使用的基准上开展的广泛实验表明，本方法在分类与分割CTTA任务中均达到最佳性能。值得注意的是，本方法可作为大型模型的新型迁移范式，在适应持续变化分布方面展现出优异效果。项目页面：https://sites.google.com/view/iclr2024-vida/home。