Self-supervised learning (SSL) learns high-quality representations from large pools of unlabeled training data. As datasets grow larger, it becomes crucial to identify the examples that contribute the most to learning such representations. This enables efficient SSL by reducing the volume of data required for learning high-quality representations. Nevertheless, quantifying the value of examples for SSL has remained an open question. In this work, we address this for the first time, by proving that examples that contribute the most to contrastive SSL are those that have the most similar augmentations to other examples, in expectation. We provide rigorous guarantees for the generalization performance of SSL on such subsets. Empirically, we discover, perhaps surprisingly, the subsets that contribute the most to SSL are those that contribute the least to supervised learning. Through extensive experiments, we show that our subsets outperform random subsets by more than 3% on CIFAR100, CIFAR10, and STL10. Interestingly, we also find that we can safely exclude 20% of examples from CIFAR100 and 40% from STL10, without affecting downstream task performance.

翻译：自监督学习（SSL）从大量无标签训练数据中学习高质量表征。随着数据集规模日益增大，识别对学习此类表征贡献最大的样本变得至关重要，这能通过减少所需数据量实现高效SSL。然而，量化样本对SSL的价值始终是一个开放性问题。本文首次通过理论证明，在期望意义上，对对比自监督学习贡献最大的样本是那些与其他样本具有最相似增广变换的样本。我们给出了此类子集上SSL泛化性能的严格保证。实验发现，令人惊讶的是，对SSL贡献最大的子集恰恰是对监督学习贡献最小的子集。通过大量实验，我们证明所提子集在CIFAR100、CIFAR10和STL10数据集上比随机子集性能提升超过3%。有趣的是，我们还发现可以安全地从CIFAR100中剔除20%的样本、从STL10中剔除40%的样本，而不影响下游任务性能。