Contrastive learning usually compares one positive anchor sample with lots of negative samples to perform Self-Supervised Learning (SSL). Alternatively, non-contrastive learning, as exemplified by methods like BYOL, SimSiam, and Barlow Twins, accomplishes SSL without the explicit use of negative samples. Inspired by the existing analysis for contrastive learning, we provide a reproducing kernel Hilbert space (RKHS) understanding of many existing non-contrastive learning methods. Subsequently, we propose a novel loss function, Kernel-SSL, which directly optimizes the mean embedding and the covariance operator within the RKHS. In experiments, our method Kernel-SSL outperforms state-of-the-art methods by a large margin on ImageNet datasets under the linear evaluation settings. Specifically, when performing 100 epochs pre-training, our method outperforms SimCLR by 4.6%.

翻译：对比学习通常通过将一个正锚点样本与大量负样本进行比较来实现自监督学习（SSL）。相比之下，非对比学习（如BYOL、SimSiam和Barlow Twins等方法）在不显式使用负样本的情况下完成SSL。受现有对比学习分析的启发，我们为许多现有的非对比学习方法提供了再生核希尔伯特空间（RKHS）视角下的理解。基于此，我们提出了一种新的损失函数Kernel-SSL，该函数直接优化RKHS中的均值嵌入和协方差算子。在实验中，我们的Kernel-SSL方法在ImageNet数据集上的线性评估设置下，以较大优势超越当前最先进的方法。具体而言，在进行100个周期的预训练时，我们的方法比SimCLR提升了4.6%。