Recent research on robustness has revealed significant performance gaps between neural image classifiers trained on datasets that are similar to the test set, and those that are from a naturally shifted distribution, such as sketches, paintings, and animations of the object categories observed during training. Prior work focuses on reducing this gap by designing engineered augmentations of training data or through unsupervised pretraining of a single large model on massive in-the-wild training datasets scraped from the Internet. However, the notion of a dataset is also undergoing a paradigm shift in recent years. With drastic improvements in the quality, ease-of-use, and access to modern generative models, generated data is pervading the web. In this light, we study the question: How do these generated datasets influence the natural robustness of image classifiers? We find that Imagenet classifiers trained on real data augmented with generated data achieve higher accuracy and effective robustness than standard training and popular augmentation strategies in the presence of natural distribution shifts. We analyze various factors influencing these results, including the choice of conditioning strategies and the amount of generated data. Additionally, we find that the standard ImageNet classifiers suffer a performance degradation of upto 20\% on the generated data, indicating their fragility at accurately classifying the objects under novel variations. Lastly, we demonstrate that the image classifiers, which have been trained on real data augmented with generated data from the base generative model, exhibit greater resilience to natural distribution shifts compared to the classifiers trained on real data augmented with generated data from the finetuned generative model on the real data. The code, models, and datasets are available at https://github.com/Hritikbansal/generative-robustness.

翻译：最近关于鲁棒性的研究揭示了，在训练数据与测试集相似的神经网络图像分类器与训练数据来自自然分布偏移（如训练中观察到的物体类别的素描、绘画和动画）的神经网络图像分类器之间，存在显著的性能差距。先前的工作侧重于通过设计训练数据的工程化增强，或通过在大规模从互联网抓取的野外训练数据集上对单个大模型进行无监督预训练，来缩小这一差距。然而，近年来，数据集的概念也正经历着范式转变。随着现代生成模型在质量、易用性和可及性方面的显著提升，生成数据正在互联网上普及。在此背景下，我们研究了一个问题：这些生成数据集如何影响图像分类器的自然鲁棒性？我们发现，与标准训练和流行增强策略相比，在自然分布偏移存在的情况下，使用真实数据与生成数据增强相结合训练的ImageNet分类器达到了更高的准确性和有效鲁棒性。我们分析了影响这些结果的各种因素，包括条件化策略的选择和生成数据的数量。此外，我们发现，标准的ImageNet分类器在生成数据上的性能下降高达20%，这表明它们在新颖变体下对物体进行分类时存在脆弱性。最后，我们证明，与使用在真实数据上微调的生成模型生成的生成数据增强的真实数据训练的分类器相比，使用来自基础生成模型的生成数据增强的真实数据训练的图像分类器对自然分布偏移表现出更强的韧性。代码、模型和数据集可在 https://github.com/Hritikbansal/generative-robustness 获取。