Given the inevitability of domain shifts during inference in real-world applications, test-time adaptation (TTA) is essential for model adaptation after deployment. However, the real-world scenario of continuously changing target distributions presents challenges including catastrophic forgetting and error accumulation. Existing TTA methods for non-stationary domain shifts, while effective, incur excessive computational load, making them impractical for on-device settings. In this paper, we introduce a layer-wise auto-weighting algorithm for continual and gradual TTA that autonomously identifies layers for preservation or concentrated adaptation. By leveraging the Fisher Information Matrix (FIM), we first design the learning weight to selectively focus on layers associated with log-likelihood changes while preserving unrelated ones. Then, we further propose an exponential min-max scaler to make certain layers nearly frozen while mitigating outliers. This minimizes forgetting and error accumulation, leading to efficient adaptation to non-stationary target distribution. Experiments on CIFAR-10C, CIFAR-100C, and ImageNet-C show our method outperforms conventional continual and gradual TTA approaches while significantly reducing computational load, highlighting the importance of FIM-based learning weight in adapting to continuously or gradually shifting target domains.

翻译：鉴于实际应用中推理阶段域偏移的不可避免性，测试时自适应（TTA）对于部署后的模型适应至关重要。然而，目标分布持续变化的现实场景带来了灾难性遗忘和误差累积等挑战。现有针对非平稳域偏移的TTA方法虽然有效，但会带来过高的计算负载，使其难以适用于设备端场景。本文提出一种用于持续渐进式TTA的逐层自动加权算法，可自主识别需要保留或集中适应的网络层。通过利用Fisher信息矩阵（FIM），我们首先设计学习权重，选择性聚焦与对数似然变化相关的层，同时保留无关层。进而提出指数化最小-最大值缩放器，使特定层近乎冻结并抑制异常值。该方法最小化遗忘与误差累积，实现对非平稳目标分布的高效适应。在CIFAR-10C、CIFAR-100C和ImageNet-C上的实验表明，本方法在显著降低计算负载的同时，性能优于传统持续与渐进式TTA方法，凸显了基于FIM的学习权重在适应持续或渐进偏移目标域中的重要性。