In this paper, we provide an information-theoretic perspective on Variance-Invariance-Covariance Regularization (VICReg) for self-supervised learning. To do so, we first demonstrate how information-theoretic quantities can be obtained for deterministic networks as an alternative to the commonly used unrealistic stochastic networks assumption. Next, we relate the VICReg objective to mutual information maximization and use it to highlight the underlying assumptions of the objective. Based on this relationship, we derive a generalization bound for VICReg, providing generalization guarantees for downstream supervised learning tasks and present new self-supervised learning methods, derived from a mutual information maximization objective, that outperform existing methods in terms of performance. This work provides a new information-theoretic perspective on self-supervised learning and Variance-Invariance-Covariance Regularization in particular and guides the way for improved transfer learning via information-theoretic self-supervised learning objectives.

翻译：本文提出了一种基于信息论的视角来理解自监督学习中的方差-不变性-协方差正则化。为此，我们首先展示了如何在确定性网络中获取信息论量，以替代常用的、不切实际的随机网络假设。接着，我们将VICReg目标与互信息最大化联系起来，并利用这一关联揭示该目标背后的隐含假设。基于这一关系，我们推导出VICReg的泛化界，为下游监督学习任务提供泛化保证，并提出基于互信息最大化目标的新自监督学习方法，其在性能上优于现有方法。本研究为自监督学习，特别是方差-不变性-协方差正则化，提供了新的信息论视角，并指导了通过信息论自监督学习目标改进迁移学习的路径。