This paper proposes a novel online evaluation protocol for Test Time Adaptation (TTA) methods, which penalizes slower methods by providing them with fewer samples for adaptation. TTA methods leverage unlabeled data at test time to adapt to distribution shifts. Although many effective methods have been proposed, their impressive performance usually comes at the cost of significantly increased computation budgets. Current evaluation protocols overlook the effect of this extra computation cost, affecting their real-world applicability. To address this issue, we propose a more realistic evaluation protocol for TTA methods, where data is received in an online fashion from a constant-speed data stream, thereby accounting for the method's adaptation speed. We apply our proposed protocol to benchmark several TTA methods on multiple datasets and scenarios. Extensive experiments show that, when accounting for inference speed, simple and fast approaches can outperform more sophisticated but slower methods. For example, SHOT from 2020, outperforms the state-of-the-art method SAR from 2023 in this setting. Our results reveal the importance of developing practical TTA methods that are both accurate and efficient.

翻译：本文提出了一种新颖的在线评估协议，用于测试时间自适应方法。该协议通过为较慢的方法提供更少的自适应样本来对其进行惩罚。TTA方法在测试时利用未标记数据来适应分布偏移。尽管已提出许多有效方法，但其令人印象深刻的性能通常以显著增加的计算预算为代价。当前的评估协议忽视了这种额外计算成本的影响，从而影响了其在现实世界中的适用性。为解决此问题，我们为TTA方法提出了一种更现实的评估协议，其中数据以在线方式从恒定速度的数据流中接收，从而考虑了方法的自适应速度。我们应用所提出的协议，在多个数据集和场景下对几种TTA方法进行基准测试。大量实验表明，当考虑推理速度时，简单快速的方法可以胜过更复杂但更慢的方法。例如，在此设置下，2020年的SHOT方法优于2023年的最先进方法SAR。我们的结果揭示了开发既准确又高效的实用TTA方法的重要性。