Understanding self-supervised learning is important but challenging. Previous theoretical works study the role of pretraining losses, and view neural networks as general black boxes. However, the recent work of Saunshi et al. argues that the model architecture -- a component largely ignored by previous works -- also has significant influences on the downstream performance of self-supervised learning. In this work, we provide the first theoretical analysis of self-supervised learning that incorporates the effect of inductive biases originating from the model class. In particular, we focus on contrastive learning -- a popular self-supervised learning method that is widely used in the vision domain. We show that when the model has limited capacity, contrastive representations would recover certain special clustering structures that are compatible with the model architecture, but ignore many other clustering structures in the data distribution. As a result, our theory can capture the more realistic setting where contrastive representations have much lower dimensionality than the number of clusters in the data distribution. We instantiate our theory on several synthetic data distributions, and provide empirical evidence to support the theory.

翻译：理解自监督学习至关重要但充满挑战。以往的理论工作侧重于研究预训练损失函数的作用，并将神经网络视为通用黑箱。然而，Saunshi等人近期的研究指出，模型架构——这一被以往工作大量忽略的组成部分——对自监督学习的下游性能也产生显著影响。在本工作中，我们首次提供了将源自模型类别的归纳偏好效应纳入考虑的自监督学习理论分析。具体而言，我们聚焦于对比学习——一种在视觉领域广泛使用的流行自监督学习方法。我们证明，当模型容量有限时，对比学习表示将恢复与模型架构兼容的特定聚类结构，而忽略数据分布中的许多其他聚类结构。因此，我们的理论能够捕捉更现实的场景，即对比学习表示的维度远低于数据分布中的聚类数。我们在若干合成数据分布上实例化了该理论，并提供了支持该理论的实证证据。