Contrastive Language-Image Pre-training (CLIP) has emerged as a simple yet effective way to train large-scale vision-language models. CLIP demonstrates impressive zero-shot classification and retrieval on diverse downstream tasks. However, to leverage its full potential, fine-tuning still appears to be necessary. Fine-tuning the entire CLIP model can be resource-intensive and unstable. Moreover, recent methods that aim to circumvent this need for fine-tuning still require access to images from the target distribution. In this paper, we pursue a different approach and explore the regime of training-free "name-only transfer" in which the only knowledge we possess about the downstream task comprises the names of downstream target categories. We propose a novel method, SuS-X, consisting of two key building blocks -- SuS and TIP-X, that requires neither intensive fine-tuning nor costly labelled data. SuS-X achieves state-of-the-art zero-shot classification results on 19 benchmark datasets. We further show the utility of TIP-X in the training-free few-shot setting, where we again achieve state-of-the-art results over strong training-free baselines. Code is available at https://github.com/vishaal27/SuS-X.

翻译：对比语言-图像预训练（CLIP）已成为一种简单而有效的训练大规模视觉语言模型的方法。CLIP在多种下游任务中展现出令人印象深刻的零样本分类和检索能力。然而，若要充分发挥其潜力，微调似乎仍不可或缺。对整个CLIP模型进行微调可能消耗大量资源且不稳定。此外，近期旨在规避微调需求的方法仍需要获取目标分布中的图像样本。本文采用不同思路，探索了无需训练的"仅名称迁移"范式——在此情境下，我们对下游任务的唯一先验知识仅包含目标类别的名称。我们提出名为SuS-X的新方法，其核心由两个关键模块——SuS与TIP-X构成，无需繁重微调或昂贵标注数据。SuS-X在19个基准数据集上达到了零样本分类的最优结果。我们进一步展示了TIP-X在无需训练的小样本场景中的效用，在此场景下再次超越强无需训练基线方法，取得最优结果。代码已开源至https://github.com/vishaal27/SuS-X。