Contrastive Language-Image Pretraining (CLIP) stands out as a prominent method for image representation learning. Various architectures, from vision transformers (ViTs) to convolutional networks (ResNets) have been trained with CLIP to serve as general solutions to diverse vision tasks. This paper explores the differences across various CLIP-trained vision backbones. Despite using the same data and training objective, we find that these architectures have notably different representations, different classification performance across datasets, and different robustness properties to certain types of image perturbations. Our findings indicate a remarkable possible synergy across backbones by leveraging their respective strengths. In principle, classification accuracy could be improved by over 40 percentage with an informed selection of the optimal backbone per test example.Using this insight, we develop a straightforward yet powerful approach to adaptively ensemble multiple backbones. The approach uses as few as one labeled example per class to tune the adaptive combination of backbones. On a large collection of datasets, the method achieves a remarkable increase in accuracy of up to 39.1% over the best single backbone, well beyond traditional ensembles

翻译：对比语言-图像预训练（CLIP）已成为图像表示学习的重要方法。从视觉Transformer（ViT）到卷积网络（ResNet）等多种架构均已通过CLIP训练，成为解决各类视觉任务的通用方案。本文探究了不同CLIP训练视觉骨干网络间的差异。尽管使用相同数据和训练目标，我们发现这些架构具有显著不同的表示特征、在不同数据集上的分类性能各异，并对特定类型的图像扰动表现出不同的鲁棒性。研究结果表明，通过利用各骨干网络的独特优势，可能实现显著的协同效应。理论上，通过为每个测试样本智能选择最优骨干网络，分类准确率可提升超过40个百分点。基于这一发现，我们开发了一种简洁而强大的自适应多骨干网络集成方法。该方法仅需每类一个标注样本即可调整骨干网络的自适应组合。在大量数据集上的实验表明，该方法相比最佳单骨干网络实现了高达39.1%的准确率提升，显著超越传统集成方法。