Contrastive Language-Image Pretraining (CLIP) stands out as a prominent method for image representation learning. Various neural architectures, spanning Transformer-based models like Vision Transformers (ViTs) to Convolutional Networks (ConvNets) like ResNets, are trained with CLIP and serve as universal backbones across diverse vision tasks. Despite utilizing the same data and training objectives, the effectiveness of representations learned by these architectures raises a critical question. Our investigation explores the differences in CLIP performance among these backbone architectures, revealing significant disparities in their classifications. Notably, normalizing these representations results in substantial performance variations. Our findings showcase a remarkable possible synergy between backbone predictions that could reach an improvement of over 20% through informed selection of the appropriate backbone. Moreover, we propose a simple, yet effective approach to combine predictions from multiple backbones, leading to a notable performance boost of up to 6.34\%. We will release the code for reproducing the results.

翻译：对比语言-图像预训练（CLIP）是一种突出的图像表征学习方法。从基于Transformer的模型（如视觉Transformer，ViTs）到基于卷积网络（ConvNets）的模型（如ResNets），各种神经架构都通过CLIP进行训练，并作为通用骨干网络服务于多样化的视觉任务。尽管使用相同的数据和训练目标，这些架构学到的表征有效性引发了一个关键问题。我们的研究探索了不同骨干网络架构在CLIP性能上的差异，揭示了其分类结果存在显著的不一致性。值得注意的是，对这些表征进行归一化处理会导致性能的显著差异。我们的发现展示了骨干网络预测之间可能存在显著的协同效应，通过明智地选择适当的骨干网络，性能可提升超过20%。此外，我们提出了一种简单而有效的方法来融合多个骨干网络的预测结果，从而实现了高达6.34%的性能提升。我们将公开发布用于复现结果的相关代码。