Pre-training has been widely adopted in deep learning to improve model performance, especially when the training data for a target task is limited. In our work, we seek to understand the implications of this training strategy on the generalization properties of downstream models. More specifically, we ask the following question: how do properties of the pre-training distribution affect the robustness of a fine-tuned model? The properties we explore include the label space, label semantics, image diversity, data domains, and data quantity of the pre-training distribution. We find that the primary factor influencing downstream effective robustness (Taori et al., 2020) is data quantity, while other factors have limited significance. For example, reducing the number of ImageNet pre-training classes by 4x while increasing the number of images per class by 4x (that is, keeping total data quantity fixed) does not impact the robustness of fine-tuned models. We demonstrate our findings on pre-training distributions drawn from various natural and synthetic data sources, primarily using the iWildCam-WILDS distribution shift as a test for downstream robustness.

翻译：预训练在深度学习中已被广泛采用以提升模型性能，尤其是在目标任务训练数据有限的情况下。本研究旨在剖析该训练策略对下游模型泛化能力的影响，具体提出以下问题：预训练分布的特性如何影响微调模型的鲁棒性？我们探索的特性包括预训练分布的标签空间、标签语义、图像多样性、数据域及数据量。研究发现，影响下游有效鲁棒性（Taori等，2020）的主要因素是数据量，而其他因素的影响相对有限。例如，将ImageNet预训练类别数量减少四倍，同时将每个类别的图像数量增加四倍（即保持总数据量不变）并不会影响微调模型的鲁棒性。我们从多种自然与合成数据源中提取预训练分布，并主要采用iWildCam-WILDS数据分布偏移作为下游鲁棒性测试，从而验证上述发现。