There are ubiquitous distribution shifts in the real world. However, deep neural networks (DNNs) are easily biased towards the training set, which causes severe performance degradation when they receive out-of-distribution data. Many methods are studied to train models that generalize under various distribution shifts in the literature of domain generalization (DG). However, the recent DomainBed and WILDS benchmarks challenged the effectiveness of these methods. Aiming at the problems in the existing research, we propose a new domain generalization task for handwritten Chinese character recognition (HCCR) to enrich the application scenarios of DG method research. We evaluate eighteen DG methods on the proposed PaHCC (Printed and Handwritten Chinese Characters) dataset and show that the performance of existing methods on this dataset is still unsatisfactory. Besides, under a designed dynamic DG setting, we reveal more properties of DG methods and argue that only the leave-one-domain-out protocol is unreliable. We advocate that researchers in the DG community refer to dynamic performance of methods for more comprehensive and reliable evaluation. Our dataset and evaluations bring new perspectives to the community for more substantial progress. We will make our dataset public with the article published to facilitate the study of domain generalization.

翻译：现实世界中普遍存在分布偏移。然而，深度神经网络（DNN）易受训练集偏置影响，导致其在处理分布外数据时出现严重性能下降。在域泛化（DG）研究领域，已有多种方法致力于训练能适应多种分布偏移的泛化模型。但近期DomainBed与WILDS基准测试对这些方法的有效性提出了挑战。针对现有研究中的问题，我们提出了一项面向手写汉字识别（HCCR）的新型域泛化任务，以丰富DG方法研究的应用场景。我们在所构建的PaHCC（印刷体与手写体汉字）数据集上评估了18种DG方法，结果表明现有方法在该数据集上的性能仍不理想。此外，在动态DG设置下，我们揭示了DG方法的更多特性，并指出仅采用留一域验证协议不可靠。我们建议DG领域的研究者应关注方法的动态性能，以实现更全面可靠的评估。本数据集与评估为领域实质性进展提供了新视角。待论文发表后，我们将公开该数据集以促进域泛化研究。