Deep learning models are challenged by the distribution shift between the training data and test data. Recently, the large models pre-trained on diverse data demonstrate unprecedented robustness to various distribution shifts. However, fine-tuning on these models can lead to a trade-off between in-distribution (ID) performance and out-of-distribution (OOD) robustness. Existing methods for tackling this trade-off do not explicitly address the OOD robustness problem. In this paper, based on causal analysis on the aforementioned problems, we propose a novel fine-tuning method, which use masked images as counterfactual samples that help improving the robustness of the fine-tuning model. Specifically, we mask either the semantics-related or semantics-unrelated patches of the images based on class activation map to break the spurious correlation, and refill the masked patches with patches from other images. The resulting counterfactual samples are used in feature-based distillation with the pre-trained model. Extensive experiments verify that regularizing the fine-tuning with the proposed masked images can achieve a better trade-off between ID and OOD, surpassing previous methods on the OOD performance. Our code will be publicly available.

翻译：深度学习模型面临训练数据与测试数据分布偏移的挑战。近年来，在多源数据上预训练的大模型展现出应对各类分布偏移的空前鲁棒性。然而，在这些模型上进行微调会导致分布内性能与分布外鲁棒性之间的权衡。现有解决该权衡的方法并未明确处理分布外鲁棒性问题。本文基于上述问题的因果分析，提出一种新型微调方法，利用掩码图像作为反事实样本以提升微调模型的鲁棒性。具体而言，我们根据类别激活图对图像中与语义相关或无关的块进行掩码操作以打破虚假关联，并用其他图像的块填补被掩码区域。生成的对抗样本与预训练模型进行基于特征的蒸馏。大量实验证明，使用所提出的掩码图像对微调过程进行正则化，能够在分布内与分布外性能间实现更优权衡，在分布外表现上超越现有方法。我们的代码将公开提供。