In recent advancements in Test Time Adaptation (TTA), most existing methodologies focus on updating normalization layers to adapt to the test domain. However, the reliance on normalization-based adaptation presents key challenges. First, normalization layers such as Batch Normalization (BN) are highly sensitive to small batch sizes, leading to unstable and inaccurate statistics. Moreover, normalization-based adaptation is inherently constrained by the structure of the pre-trained model, as it relies on training-time statistics that may not generalize well to unseen domains. These issues limit the effectiveness of normalization-based TTA approaches, especially under significant domain shift. In this paper, we introduce a novel paradigm based on the concept of a Buffer layer, which addresses the fundamental limitations of normalization layer updates. Unlike existing methods that modify the core parameters of the model, our approach preserves the integrity of the pre-trained backbone, inherently mitigating the risk of catastrophic forgetting during online adaptation. Through comprehensive experimentation, we demonstrate that our approach not only outperforms traditional methods in mitigating domain shift and enhancing model robustness, but also exhibits strong resilience to forgetting. Furthermore, our Buffer layer is modular and can be seamlessly integrated into nearly all existing TTA frameworks, resulting in consistent performance improvements across various architectures. These findings validate the effectiveness and versatility of the proposed solution in real-world domain adaptation scenarios. The code is available at https://github.com/hyeongyu-kim/Buffer_TTA.

翻译：在测试时适应（TTA）的最新进展中，现有方法大多侧重于更新归一化层以适应测试域。然而，基于归一化的适应方法存在关键挑战。首先，如批归一化（BN）等归一化层对小批量大小高度敏感，导致统计量不稳定且不准确。此外，基于归一化的适应本质上受限于预训练模型的结构，因为它依赖于训练时的统计量，而这些统计量可能无法很好地泛化到未见过的域。这些问题限制了基于归一化的TTA方法的有效性，尤其是在显著的域偏移下。本文中，我们引入了一种基于缓冲层概念的新范式，以解决归一化层更新的根本局限性。与修改模型核心参数的现有方法不同，我们的方法保持了预训练骨干网络的完整性，从而在在线适应过程中本质上减轻了灾难性遗忘的风险。通过全面的实验，我们证明我们的方法不仅在缓解域偏移和增强模型鲁棒性方面优于传统方法，而且表现出强大的抗遗忘能力。此外，我们的缓冲层是模块化的，可以无缝集成到几乎所有现有的TTA框架中，从而在各种架构上带来一致的性能提升。这些发现验证了所提解决方案在现实世界域适应场景中的有效性和通用性。代码可在 https://github.com/hyeongyu-kim/Buffer_TTA 获取。