Recently, large-scale pre-trained Vision and Language (VL) models have set a new state-of-the-art (SOTA) in zero-shot visual classification enabling open-vocabulary recognition of potentially unlimited set of categories defined as simple language prompts. However, despite these great advances, the performance of these zeroshot classifiers still falls short of the results of dedicated (closed category set) classifiers trained with supervised fine tuning. In this paper we show, for the first time, how to reduce this gap without any labels and without any paired VL data, using an unlabeled image collection and a set of texts auto-generated using a Large Language Model (LLM) describing the categories of interest and effectively substituting labeled visual instances of those categories. Using our label-free approach, we are able to attain significant performance improvements over the zero-shot performance of the base VL model and other contemporary methods and baselines on a wide variety of datasets, demonstrating absolute improvement of up to 11.7% (3.8% on average) in the label-free setting. Moreover, despite our approach being label-free, we observe 1.3% average gains over leading few-shot prompting baselines that do use 5-shot supervision.

翻译：近期，大规模预训练的视觉语言模型在零样本视觉分类任务中取得了最新最优结果，通过简单的语言提示即可实现潜在无限类别集合的开放词汇识别。然而，尽管取得显著进展，这些零样本分类器的性能仍低于经过监督微调的专用（封闭类别集）分类器。本文首次证明，如何在不依赖任何标签或配对视觉语言数据的情况下，通过利用无标签图像集合和大型语言模型自动生成的描述目标类别的文本集合，有效替代这些类别的标注视觉实例来缩小这一差距。采用我们的免标签方法，能够在多种数据集上显著提升基础视觉语言模型及其他当代方法与基线的零样本性能，在免标签设定下实现最高11.7%（平均3.8%）的绝对改进。更值得注意的是，尽管我们的方法无需标注，相较于使用5-shot监督的领先少样本提示基线，仍可获得1.3%的平均性能增益。