Given the inevitability of domain shifts during inference in real-world applications, test-time adaptation (TTA) is essential for model adaptation after deployment. However, the real-world scenario of continuously changing target distributions presents challenges including catastrophic forgetting and error accumulation. Existing TTA methods for non-stationary domain shifts, while effective, incur excessive computational load, making them impractical for on-device settings. In this paper, we introduce a layer-wise auto-weighting algorithm for continual and gradual TTA that autonomously identifies layers for preservation or concentrated adaptation. By leveraging the Fisher Information Matrix (FIM), we first design the learning weight to selectively focus on layers associated with log-likelihood changes while preserving unrelated ones. Then, we further propose an exponential min-max scaler to make certain layers nearly frozen while mitigating outliers. This minimizes forgetting and error accumulation, leading to efficient adaptation to non-stationary target distribution. Experiments on CIFAR-10C, CIFAR-100C, and ImageNet-C show our method outperforms conventional continual and gradual TTA approaches while significantly reducing computational load, highlighting the importance of FIM-based learning weight in adapting to continuously or gradually shifting target domains.

翻译：鉴于实际应用中推理时域偏移的不可避免性，测试时自适应（test-time adaptation, TTA）对于部署后的模型调整至关重要。然而，真实场景中目标分布持续变化会带来灾难性遗忘和误差累积等挑战。现有应对非平稳域偏移的TTA方法虽有效，但计算负载过高，难以适用于设备端场景。本文提出一种面向连续与渐进式TTA的逐层自动加权算法，可自主识别需要保留或针对性调整的网络层。首先，我们利用Fisher信息矩阵（FIM）设计学习权重，选择性关注与对数似然变化相关的层，同时保留无关层。其次，进一步提出指数最小-最大缩放器，在抑制异常值的同时使特定层近乎冻结。该方法能最小化遗忘与误差累积，从而高效适应非平稳目标分布。在CIFAR-10C、CIFAR-100C和ImageNet-C上的实验表明，本方法在显著降低计算负载的同时，性能优于传统连续与渐进式TTA方法，突显了基于FIM的学习权重在适应连续或渐进变化的目标域中的重要性。