The fusion of vision and language has brought about a transformative shift in computer vision through the emergence of Vision-Language Models (VLMs). However, the resource-intensive nature of existing VLMs poses a significant challenge. We need an accessible method for developing the next generation of VLMs. To address this issue, we propose Zoom-shot, a novel method for transferring the zero-shot capabilities of CLIP to any pre-trained vision encoder. We do this by exploiting the multimodal information (i.e. text and image) present in the CLIP latent space through the use of specifically designed multimodal loss functions. These loss functions are (1) cycle-consistency loss and (2) our novel prompt-guided knowledge distillation loss (PG-KD). PG-KD combines the concept of knowledge distillation with CLIP's zero-shot classification, to capture the interactions between text and image features. With our multimodal losses, we train a $\textbf{linear mapping}$ between the CLIP latent space and the latent space of a pre-trained vision encoder, for only a $\textbf{single epoch}$. Furthermore, Zoom-shot is entirely unsupervised and is trained using $\textbf{unpaired}$ data. We test the zero-shot capabilities of a range of vision encoders augmented as new VLMs, on coarse and fine-grained classification datasets, outperforming the previous state-of-the-art in this problem domain. In our ablations, we find Zoom-shot allows for a trade-off between data and compute during training; and our state-of-the-art results can be obtained by reducing training from 20% to 1% of the ImageNet training data with 20 epochs. All code and models are available on GitHub.

翻译：视觉与语言的融合通过视觉语言模型（VLM）的出现为计算机视觉带来了变革性转变。然而现有VLM的资源密集型特性构成了重大挑战，我们需要一种易于实现的方法来开发下一代VLM。针对此问题，我们提出Zoom-shot——一种将CLIP零样本能力迁移至任意预训练视觉编码器的新方法。该方法通过利用专门设计的**多模态损失函数**，在CLIP隐含空间中挖掘多模态信息（即文本与图像）。这些损失函数包括：(1) 循环一致性损失；以及(2) 我们提出的新型**提示引导知识蒸馏损失**（PG-KD）。PG-KD将知识蒸馏概念与CLIP的零样本分类相结合，以捕获文本与图像特征之间的交互。通过多模态损失，我们训练CLIP隐含空间与预训练视觉编码器隐含空间之间的**线性映射**，且仅需**单个周期**。此外，Zoom-shot完全无监督，并使用**非配对**数据进行训练。我们在粗粒度与细粒度分类数据集上测试了增强为新VLM的多类视觉编码器的零样本能力，结果均超越该问题领域的前沿方法。消融实验表明，Zoom-shot可在训练时实现数据与计算量之间的权衡；通过将ImageNet训练数据从20%减少至1%并采用20个训练周期，我们即可获得最优结果。所有代码与模型均已开源至GitHub。