Test-time adaptation (TTA) is a task that continually adapts a pre-trained source model to the target domain during inference. One popular approach involves fine-tuning model with cross-entropy loss according to estimated pseudo-labels. However, its performance is significantly affected by noisy pseudo-labels. This study reveals that minimizing the classification error of each sample causes the cross-entropy loss's vulnerability to label noise. To address this issue, we propose a novel Decoupled Prototype Learning (DPL) method that features prototype-centric loss computation. First, we decouple the optimization of class prototypes. For each class prototype, we reduce its distance with positive samples and enlarge its distance with negative samples in a contrastive manner. This strategy prevents the model from overfitting to noisy pseudo-labels. Second, we propose a memory-based strategy to enhance DPL's robustness for the small batch sizes often encountered in TTA. We update each class's pseudo-feature from a memory in a momentum manner and insert an additional DPL loss. Finally, we introduce a consistency regularization-based approach to leverage samples with unconfident pseudo-labels. This approach transfers feature styles of samples with unconfident pseudo-labels to those with confident pseudo-labels. Thus, more reliable samples for TTA are created. The experimental results demonstrate that our methods achieve state-of-the-art performance on domain generalization benchmarks, and reliably improve the performance of self-training-based methods on image corruption benchmarks. The code will be released.

翻译：测试时自适应（TTA）是一项在推理过程中持续将预训练源模型适配到目标域的任务。一种常见方法根据估计的伪标签使用交叉熵损失微调模型。然而，其性能受噪声伪标签的影响显著。本研究表明，最小化每个样本的分类误差会导致交叉熵损失对标签噪声的脆弱性。为解决该问题，我们提出一种新颖的解耦原型学习（DPL）方法，其特征是以原型为中心的损失计算。首先，我们解耦类原型的优化。对于每个类原型，我们以对比方式缩小其与正样本的距离，并扩大其与负样本的距离。该策略防止模型过拟合到噪声伪标签。其次，我们提出基于记忆库的策略，以增强DPL在TTA常见小批量大小下的鲁棒性。我们以动量方式从记忆库更新每个类的伪特征，并插入额外的DPL损失。最后，我们引入基于一致性正则化的方法，利用非置信伪标签的样本。该方法将非置信伪标签样本的特征风格迁移至置信伪标签样本，从而为TTA生成更可靠的样本。实验结果表明，我们的方法在域泛化基准上达到最新性能，并可靠提升了基于自训练方法在图像损坏基准上的表现。代码将开源。