Test-Time Adaptation (TTA) has recently emerged as a promising approach for tackling the robustness challenge under distribution shifts. However, the lack of consistent settings and systematic studies in prior literature hinders thorough assessments of existing methods. To address this issue, we present TTAB, a test-time adaptation benchmark that encompasses ten state-of-the-art algorithms, a diverse array of distribution shifts, and two evaluation protocols. Through extensive experiments, our benchmark reveals three common pitfalls in prior efforts. First, selecting appropriate hyper-parameters, especially for model selection, is exceedingly difficult due to online batch dependency. Second, the effectiveness of TTA varies greatly depending on the quality and properties of the model being adapted. Third, even under optimal algorithmic conditions, none of the existing methods are capable of addressing all common types of distribution shifts. Our findings underscore the need for future research in the field to conduct rigorous evaluations on a broader set of models and shifts, and to re-examine the assumptions behind the empirical success of TTA. Our code is available at \url{https://github.com/lins-lab/ttab}.

翻译：测试时适应（Test-Time Adaptation, TTA）近期成为一种应对分布偏移下鲁棒性挑战的有前景方法。然而，现有文献中缺乏一致的设置和系统性研究，阻碍了对现有方法的全面评估。针对这一问题，我们提出了TTAB——一个测试时适应基准，包含十种最先进算法、多种分布偏移类型以及两种评估协议。通过大量实验，我们的基准揭示了先前研究中的三个常见陷阱。首先，由于在线批次依赖性，选择合适的超参数（尤其是模型选择的超参数）极为困难。其次，TTA的有效性在很大程度上取决于被适应模型的质量和特性。第三，即使在最优算法条件下，现有方法也无法应对所有常见类型的分布偏移。我们的研究结果强调，该领域未来研究需要在更广泛的模型和偏移类型上进行严格评估，并重新审视TTA实证成功背后的假设。我们的代码可于\url{https://github.com/lins-lab/ttab}获取。