This paper proposes a novel online evaluation protocol for Test Time Adaptation (TTA) methods, which penalizes slower methods by providing them with fewer samples for adaptation. TTA methods leverage unlabeled data at test time to adapt to distribution shifts. Though many effective methods have been proposed, their impressive performance usually comes at the cost of significantly increased computation budgets. Current evaluation protocols overlook the effect of this extra computation cost, affecting their real-world applicability. To address this issue, we propose a more realistic evaluation protocol for TTA methods, where data is received in an online fashion from a constant-speed data stream, thereby accounting for the method's adaptation speed. We apply our proposed protocol to benchmark several TTA methods on multiple datasets and scenarios. Extensive experiments shows that, when accounting for inference speed, simple and fast approaches can outperform more sophisticated but slower methods. For example, SHOT from 2020 outperforms the state-of-the-art method SAR from 2023 under our online setting. Our online evaluation protocol emphasizes the need for developing TTA methods that are efficient and applicable in realistic settings.

翻译：本文提出了一种新颖的在线评估协议，用于测试时间自适应（TTA）方法。该协议通过向较慢的方法提供更少的自适应样本对其进行惩罚。TTA方法利用测试时的无标签数据来适应分布偏移。尽管已有许多有效的方法被提出，但它们令人印象深刻的性能往往以显著增加的计算预算为代价。当前的评估协议忽略了这一额外计算成本的影响，从而影响了其在实际场景中的适用性。为解决这一问题，我们提出了一种更符合实际的TTA方法评估协议，其中数据以恒定速率的数据流在线接收，从而考虑了方法的自适应速度。我们应用所提出的协议在多个数据集和场景下对几种TTA方法进行了基准测试。大量实验表明，当考虑推理速度时，简单快速的方法可以胜过复杂但较慢的方法。例如，2020年的SHOT方法在我们的在线设置下优于2023年的最新方法SAR。我们的在线评估协议强调了开发在现实场景中高效且适用的TTA方法的必要性。