Since real-world machine systems are running in non-stationary and continually changing environments, Continual Test-Time Adaptation (CTTA) task is proposed to adapt the pre-trained model to continually changing target domains. Recently, existing methods mainly focus on model-based adaptation, which aims to leverage a self-training manner to extract the target domain knowledge. However, pseudo labels can be noisy and the updated model parameters are uncertain under dynamic data distributions, leading to error accumulation and catastrophic forgetting in the continual adaptation process. To tackle these challenges and maintain the model plasticity, we tactfully design a Visual Domain Adapter (ViDA) for CTTA, explicitly handling both domain-specific and domain-agnostic knowledge. Specifically, we first comprehensively explore the different domain representations of the adapters with trainable high and low-rank embedding space. Then we inject ViDAs into the pre-trained model, which leverages high-rank and low-rank prototypes to adapt the current domain distribution and maintain the continual domain-shared knowledge, respectively. To adapt to the various distribution shifts of each sample in target domains, we further propose a Homeostatic Knowledge Allotment (HKA) strategy, which adaptively merges knowledge from each ViDA with different rank prototypes. Extensive experiments conducted on four widely-used benchmarks demonstrate that our proposed method achieves state-of-the-art performance in both classification and segmentation CTTA tasks. In addition, our method can be regarded as a novel transfer paradigm and showcases promising results in zero-shot adaptation of foundation models to continual downstream tasks and distributions.

翻译：由于现实世界中的机器系统运行在非平稳且持续变化的环境中，持续测试时适应（CTTA）任务旨在使预训练模型适应持续变化的目标域。当前现有方法主要关注基于模型的适应，即通过自训练方式提取目标域知识。然而，在动态数据分布下，伪标签可能包含噪声且更新后的模型参数具有不确定性，这会导致持续适应过程中出现误差累积和灾难性遗忘。为应对这些挑战并保持模型的可塑性，我们巧妙设计了适用于CTTA的视觉域适配器（ViDA），显式处理领域特定知识与领域无关知识。具体而言，我们首先全面探索了具有可训练高秩和低秩嵌入空间的适配器在不同域表征上的差异。然后，我们将ViDA注入预训练模型，分别利用高秩原型适应当前域分布，以及低秩原型维护持续共享的领域知识。为适应目标域中每个样本的多样分布偏移，我们进一步提出稳态知识分配（HKA）策略，该策略可自适应地融合来自不同秩原型ViDA的知识。在四个广泛使用的基准数据集上进行的实验表明，所提方法在分类和分割CTTA任务中均达到最先进性能。此外，该方法可被视为一种新型迁移范式，在将基础模型零样本适应至持续下游任务与分布中展现出显著潜力。