Contrastive Language-Image Pre-training (CLIP) has emerged as a simple yet effective way to train large-scale vision-language models. CLIP demonstrates impressive zero-shot classification and retrieval on diverse downstream tasks. However, to leverage its full potential, fine-tuning still appears to be necessary. Fine-tuning the entire CLIP model can be resource-intensive and unstable. Moreover, recent methods that aim to circumvent this need for fine-tuning still require access to images from the target distribution. In this paper, we pursue a different approach and explore the regime of training-free "name-only transfer" in which the only knowledge we possess about the downstream task comprises the names of downstream target categories. We propose a novel method, SuS-X, consisting of two key building blocks -- SuS and TIP-X, that requires neither intensive fine-tuning nor costly labelled data. SuS-X achieves state-of-the-art zero-shot classification results on 19 benchmark datasets. We further show the utility of TIP-X in the training-free few-shot setting, where we again achieve state-of-the-art results over strong training-free baselines. Code is available at https://github.com/vishaal27/SuS-X.

翻译：对比语言-图像预训练（CLIP）已成为训练大规模视觉语言模型简单而有效的方法。CLIP在多种下游任务中展现出令人印象深刻的零样本分类与检索能力，但为充分利用其潜力，微调仍不可或缺。然而，对CLIP模型进行整体微调既耗费资源又不稳定。此外，近期旨在规避微调需求的方法仍需访问目标分布的图像数据。本文另辟蹊径，探索无需训练的"仅名称迁移"范式——在该范式下，我们对下游任务所掌握的全部知识仅为目标类别的名称。我们提出SuS-X这一创新方法，由SuS和TIP-X两大核心模块构成，该方法既无需密集微调，也不依赖昂贵标注数据。SuS-X在19个基准数据集上实现了零样本分类的最优性能。我们进一步展示了TIP-X在免训练小样本场景中的应用价值，同样在强免训练基线上取得了最优结果。代码已开源至https://github.com/vishaal27/SuS-X。