Variance-Invariance-Covariance Regularization (VICReg) is a self-supervised learning (SSL) method that has shown promising results on a variety of tasks. However, the fundamental mechanisms underlying VICReg remain unexplored. In this paper, we present an information-theoretic perspective on the VICReg objective. We begin by deriving information-theoretic quantities for deterministic networks as an alternative to unrealistic stochastic network assumptions. We then relate the optimization of the VICReg objective to mutual information optimization, highlighting underlying assumptions and facilitating a constructive comparison with other SSL algorithms and derive a generalization bound for VICReg, revealing its inherent advantages for downstream tasks. Building on these results, we introduce a family of SSL methods derived from information-theoretic principles that outperform existing SSL techniques.

翻译：方差-不变性-协方差正则化（VICReg）是一种自监督学习方法，在多种任务上展现了良好效果。然而，VICReg的基本机制尚未得到深入探索。本文从信息论视角阐释VICReg目标函数。我们首先为确定性网络推导信息论量，以替代不切实际的随机网络假设。进而将VICReg目标优化与互信息优化相关联，揭示其潜在假设并促进与其他SSL算法的建设性比较，同时推导VICReg的泛化界，阐明其在下游任务中的固有优势。基于这些结果，我们提出一系列源于信息论原理的自监督学习方法，其性能优于现有SSL技术。