In this paper, we provide an information-theoretic perspective on Variance-Invariance-Covariance Regularization (VICReg) for self-supervised learning. To do so, we first demonstrate how information-theoretic quantities can be obtained for deterministic networks as an alternative to the commonly used unrealistic stochastic networks assumption. Next, we relate the VICReg objective to mutual information maximization and use it to highlight the underlying assumptions of the objective. Based on this relationship, we derive a generalization bound for VICReg, providing generalization guarantees for downstream supervised learning tasks and present new self-supervised learning methods, derived from a mutual information maximization objective, that outperform existing methods in terms of performance. This work provides a new information-theoretic perspective on self-supervised learning and Variance-Invariance-Covariance Regularization in particular and guides the way for improved transfer learning via information-theoretic self-supervised learning objectives.

翻译：在本文中，我们为自监督学习中的方差-不变性-协方差正则化（VICReg）提供了信息论视角。为此，我们首先展示了如何为确定性网络获得信息论量，以替代常用的不切实际的随机网络假设。接着，我们将VICReg目标与互信息最大化相关联，并利用该关系揭示该目标的内在假设。基于这种关联，我们推导出VICReg的泛化界，为下游监督学习任务提供泛化保证，并提出新的自监督学习方法，这些方法源于互信息最大化目标，在性能上优于现有方法。本文为自监督学习（尤其是方差-不变性-协方差正则化）提供了新的信息论视角，并为通过信息论自监督学习目标改进迁移学习指明了方向。