Self-Supervised Learning (SSL) is a paradigm that leverages unlabeled data for model training. Empirical studies show that SSL can achieve promising performance in distribution shift scenarios, where the downstream and training distributions differ. However, the theoretical understanding of its transferability remains limited. In this paper, we develop a theoretical framework to analyze the transferability of self-supervised contrastive learning, by investigating the impact of data augmentation on it. Our results reveal that the downstream performance of contrastive learning depends largely on the choice of data augmentation. Moreover, we show that contrastive learning fails to learn domain-invariant features, which limits its transferability. Based on these theoretical insights, we propose a novel method called Augmentation-robust Contrastive Learning (ArCL), which guarantees to learn domain-invariant features and can be easily integrated with existing contrastive learning algorithms. We conduct experiments on several datasets and show that ArCL significantly improves the transferability of contrastive learning.

翻译：自监督学习（SSL）是一种利用无标签数据进行模型训练的范式。实证研究表明，在下游分布与训练分布不同的分布偏移场景中，SSL能够取得有前景的性能表现。然而，关于其可迁移性的理论理解仍十分有限。本文构建了一个理论框架，通过探究数据增强对自监督对比学习的影响，来分析其可迁移性。研究结果表明，对比学习的下游性能在很大程度上取决于数据增强的选择。此外，我们揭示了对比学习无法学习到领域不变特征，这限制了其可迁移性。基于这些理论洞见，我们提出了一种名为增强鲁棒对比学习（ArCL）的新方法，该方法能够保证学习到领域不变特征，并可轻松与现有对比学习算法集成。我们在多个数据集上进行了实验，结果表明ArCL显著提升了对比学习的可迁移性。