Deep learning models are challenged by the distribution shift between the training data and test data. Recently, the large models pre-trained on diverse data demonstrate unprecedented robustness to various distribution shifts. However, fine-tuning on these models can lead to a trade-off between in-distribution (ID) performance and out-of-distribution (OOD) robustness. Existing methods for tackling this trade-off do not explicitly address the OOD robustness problem. In this paper, based on causal analysis on the aforementioned problems, we propose a novel fine-tuning method, which use masked images as counterfactual samples that help improving the robustness of the fine-tuning model. Specifically, we mask either the semantics-related or semantics-unrelated patches of the images based on class activation map to break the spurious correlation, and refill the masked patches with patches from other images. The resulting counterfactual samples are used in feature-based distillation with the pre-trained model. Extensive experiments verify that regularizing the fine-tuning with the proposed masked images can achieve a better trade-off between ID and OOD performance, surpassing previous methods on the OOD performance. Our code will be publicly available.

翻译：深度学习模型面临训练数据与测试数据之间分布偏移的挑战。近年来，基于多样化数据预训练的大规模模型展现出对各种分布偏移前所未有的鲁棒性。然而，对这些模型进行微调可能导致分布内性能与分布外鲁棒性之间的权衡。现有解决该权衡问题的方法并未明确处理分布外鲁棒性问题。本文基于对上述问题的因果分析，提出一种新颖的微调方法，利用掩膜图像作为反事实样本来提升微调模型的鲁棒性。具体而言，我们基于类激活映射掩膜图像中与语义相关或无关的图块以打破虚假关联，并用其他图像的图块重新填充掩膜区域。由此生成的反事实样本被用于与预训练模型的特征蒸馏。大量实验证明，使用所提出的掩膜图像正则化微调过程，可以在分布内与分布外性能之间实现更优的权衡，在分布外性能上超越先前方法。我们的代码将公开提供。