Domain generalization aims to solve the challenge of Out-of-Distribution (OOD) generalization by leveraging common knowledge learned from multiple training domains to generalize to unseen test domains. To accurately evaluate the OOD generalization ability, it is necessary to ensure that test data information is unavailable. However, the current domain generalization protocol may still have potential test data information leakage. This paper examines the potential risks of test data information leakage in two aspects of the current protocol: pretraining on ImageNet and oracle model selection. We propose that training from scratch and using multiple test domains would result in a more precise evaluation of OOD generalization ability. We also rerun the algorithms with the modified protocol and introduce a new leaderboard to encourage future research in domain generalization with a fairer comparison.

翻译：领域泛化旨在解决分布外（OOD）泛化的挑战，通过利用从多个训练域中学习到的共同知识来泛化到未见过的测试域。为了准确评估OOD泛化能力，必须确保测试数据信息不可用。然而，当前的领域泛化协议仍可能存在潜在的测试数据信息泄露。本文考察了当前协议中两个方面的测试数据信息泄露风险：在ImageNet上的预训练以及基于Oracle的模型选择。我们提出，从零开始训练并使用多个测试域将更精确地评估OOD泛化能力。我们还通过修改后的协议重新运行算法，并引入新的排行榜，以鼓励未来在更公平的比较下进行领域泛化研究。