Language-supervised vision models have recently attracted great attention in computer vision. A common approach to build such models is to use contrastive learning on paired data across the two modalities, as exemplified by Contrastive Language-Image Pre-Training (CLIP). In this paper, under linear representation settings, (i) we initiate the investigation of a general class of nonlinear loss functions for multimodal contrastive learning (MMCL) including CLIP loss and show its connection to singular value decomposition (SVD). Namely, we show that each step of loss minimization by gradient descent can be seen as performing SVD on a contrastive cross-covariance matrix. Based on this insight, (ii) we analyze the performance of MMCL. We quantitatively show that the feature learning ability of MMCL can be better than that of unimodal contrastive learning applied to each modality even under the presence of wrongly matched pairs. This characterizes the robustness of MMCL to noisy data. Furthermore, when we have access to additional unpaired data, (iii) we propose a new MMCL loss that incorporates additional unpaired datasets. We show that the algorithm can detect the ground-truth pairs and improve performance by fully exploiting unpaired datasets. The performance of the proposed algorithm was verified by numerical experiments.

翻译：语言监督的视觉模型近年来在计算机视觉领域引起了广泛关注。构建此类模型的常见方法是对跨模态配对数据使用对比学习，如对比语言-图像预训练(CLIP)所示。本文在线性表示设置下，(i)我们首次研究了一类包含CLIP损失的多模态对比学习(MMCL)非线性损失函数，并揭示了其与奇异值分解(SVD)的联系。具体而言，我们证明了通过梯度下降进行损失最小化的每一步可视为对对比交叉协方差矩阵执行SVD。基于这一发现，(ii)我们分析了MMCL的性能。我们定量表明，即使在存在错误配对样本的情况下，MMCL的特征学习能力仍可能优于对各模态分别应用单模态对比学习。这刻画了MMCL对噪声数据的鲁棒性。此外，当能够获取额外非配对数据时，(iii)我们提出了一种整合非配对数据集的新型MMCL损失函数。结果表明，该算法能检测真实配对并通过充分利用非配对数据集提升性能。数值实验验证了所提算法的有效性。