In recent advancements in Test Time Adaptation (TTA), most existing methodologies focus on updating normalization layers to adapt to the test domain. However, the reliance on normalization-based adaptation presents key challenges. First, normalization layers such as Batch Normalization (BN) are highly sensitive to small batch sizes, leading to unstable and inaccurate statistics. Moreover, normalization-based adaptation is inherently constrained by the structure of the pre-trained model, as it relies on training-time statistics that may not generalize well to unseen domains. These issues limit the effectiveness of normalization-based TTA approaches, especially under significant domain shift. In this paper, we introduce a novel paradigm based on the concept of a Buffer layer, which addresses the fundamental limitations of normalization layer updates. Unlike existing methods that modify the core parameters of the model, our approach preserves the integrity of the pre-trained backbone, inherently mitigating the risk of catastrophic forgetting during online adaptation. Through comprehensive experimentation, we demonstrate that our approach not only outperforms traditional methods in mitigating domain shift and enhancing model robustness, but also exhibits strong resilience to forgetting. Furthermore, our Buffer layer is modular and can be seamlessly integrated into nearly all existing TTA frameworks, resulting in consistent performance improvements across various architectures. These findings validate the effectiveness and versatility of the proposed solution in real-world domain adaptation scenarios. The code is available at https://github.com/hyeongyu-kim/Buffer_TTA.

翻译：在测试时适应（TTA）的最新进展中，现有方法大多聚焦于更新归一化层以适应测试域。然而，基于归一化的适应方法存在关键挑战。首先，批归一化（BN）等归一化层对小批量数据高度敏感，导致统计量不稳定且不准确。此外，基于归一化的适应本质上受预训练模型结构的限制，因其依赖于训练时的统计量，这些统计量可能无法很好地泛化到未见过的域。这些问题限制了基于归一化的TTA方法的有效性，尤其是在显著域偏移的情况下。本文提出了一种基于缓冲层概念的新范式，以解决归一化层更新的根本局限性。与修改模型核心参数的现有方法不同，我们的方法保持了预训练主干网络的完整性，从本质上缓解了在线适应过程中灾难性遗忘的风险。通过全面的实验，我们证明该方法不仅在缓解域偏移和增强模型鲁棒性方面优于传统方法，还展现出强大的抗遗忘能力。此外，我们的缓冲层具有模块化特性，可无缝集成到几乎所有现有的TTA框架中，从而在各种架构上实现一致的性能提升。这些发现验证了所提方案在现实域适应场景中的有效性和通用性。代码可在 https://github.com/hyeongyu-kim/Buffer_TTA 获取。