Contrastive learning is an efficient approach to self-supervised representation learning. Although recent studies have made progress in the theoretical understanding of contrastive learning, the investigation of how to characterize the clusters of the learned representations is still limited. In this paper, we aim to elucidate the characterization from theoretical perspectives. To this end, we consider a kernel-based contrastive learning framework termed Kernel Contrastive Learning (KCL), where kernel functions play an important role when applying our theoretical results to other frameworks. We introduce a formulation of the similarity structure of learned representations by utilizing a statistical dependency viewpoint. We investigate the theoretical properties of the kernel-based contrastive loss via this formulation. We first prove that the formulation characterizes the structure of representations learned with the kernel-based contrastive learning framework. We show a new upper bound of the classification error of a downstream task, which explains that our theory is consistent with the empirical success of contrastive learning. We also establish a generalization error bound of KCL. Finally, we show a guarantee for the generalization ability of KCL to the downstream classification task via a surrogate bound.

翻译：对比学习是一种高效的自监督表示学习方法。尽管近期研究在对比学习的理论理解方面取得了进展，但对如何刻画所学表示聚类的研究仍十分有限。本文旨在从理论视角阐明这一刻画机制。为此，我们考虑一个基于核函数的对比学习框架——核对比学习（KCL），其中核函数在将理论结果推广至其他框架时发挥重要作用。我们通过利用统计依赖性的观点，引入了所学表示相似性结构的公式化表达。基于该公式化表达，我们研究了核对比损失的理论性质。首先证明该公式化表达刻画了基于核函数的对比学习框架所学表示的结构。本文给出了下游任务分类误差的新上界，表明我们的理论与对比学习的实证成功一致。我们还建立了KCL的泛化误差界。最后，通过代理界展示了KCL对下游分类任务泛化能力的保证。