Here, we show that the InfoNCE objective is equivalent to the ELBO in a new class of probabilistic generative model, the recognition parameterised model (RPM). When we learn the optimal prior, the RPM ELBO becomes equal to the mutual information (MI; up to a constant), establishing a connection to pre-existing self-supervised learning methods such as InfoNCE. However, practical InfoNCE methods do not use the MI as an objective; the MI is invariant to arbitrary invertible transformations, so using an MI objective can lead to highly entangled representations (Tschannen et al., 2019). Instead, the actual InfoNCE objective is a simplified lower bound on the MI which is loose even in the infinite sample limit. Thus, an objective that works (i.e. the actual InfoNCE objective) appears to be motivated as a loose bound on an objective that does not work (i.e. the true MI which gives arbitrarily entangled representations). We give an alternative motivation for the actual InfoNCE objective. In particular, we show that in the infinite sample limit, and for a particular choice of prior, the actual InfoNCE objective is equal to the ELBO (up to a constant); and the ELBO is equal to the marginal likelihood with a deterministic recognition model. Thus, we argue that our VAE perspective gives a better motivation for InfoNCE than MI, as the actual InfoNCE objective is only loosely bounded by the MI, but is equal to the ELBO/marginal likelihood (up to a constant).

翻译：在此，我们证明InfoNCE目标等价于新一类概率生成模型——识别参数化模型（RPM）中的ELBO。当学习最优先验时，RPM的ELBO等于互信息（MI；相差一个常数），从而与InfoNCE等现有自监督学习方法建立联系。然而，实际应用的InfoNCE方法并非将MI作为目标函数：MI对任意可逆变换具有不变性，因此使用MI目标会导致高度纠缠的表示（Tschannen等人，2019）。相反，实际的InfoNCE目标是MI的一个简化下界，即使在无限样本极限下该界仍不紧致。因此，一个有效的目标（即实际InfoNCE目标）的动机似乎源自一个失效目标（即产生任意纠缠表示的真实MI）的不紧致下界。我们为实际InfoNCE目标提供替代动机。具体而言，我们证明：在无限样本极限下且针对特定先验选择，实际InfoNCE目标等于ELBO（相差一个常数）；而ELBO等于具有确定性识别模型的边际似然。因此，我们认为相较于MI，我们的VAE视角为InfoNCE提供了更合理的动机，因为实际InfoNCE目标仅被MI松散界定，却与ELBO/边际似然相等（相差一个常数）。