Recent vision-language models outperform vision-only models on many image classification tasks. However, because of the absence of paired text/image descriptions, it remains difficult to fine-tune these models for fine-grained image classification. In this work, we propose a method, GIST, for generating image-specific fine-grained text descriptions from image-only datasets, and show that these text descriptions can be used to improve classification. Key parts of our method include 1. prompting a pretrained large language model with domain-specific prompts to generate diverse fine-grained text descriptions for each class and 2. using a pretrained vision-language model to match each image to label-preserving text descriptions that capture relevant visual features in the image. We demonstrate the utility of GIST by fine-tuning vision-language models on the image-and-generated-text pairs to learn an aligned vision-language representation space for improved classification. We evaluate our learned representation space in full-shot and few-shot scenarios across four diverse fine-grained classification datasets, each from a different domain. Our method achieves an average improvement of $4.1\%$ in accuracy over CLIP linear probes and an average of $1.1\%$ improvement in accuracy over the previous state-of-the-art image-text classification method on the full-shot datasets. Our method achieves similar improvements across few-shot regimes. Code is available at https://github.com/emu1729/GIST.

翻译：摘要：近期视觉-语言模型在多项图像分类任务上超越了纯视觉模型。然而，由于缺乏配对的文本/图像描述，这些模型在细粒度图像分类上的微调仍面临困难。本文提出一种名为GIST的方法，可从纯图像数据集中生成图像特定的细粒度文本描述，并证明这些文本描述可用于提升分类性能。本方法的关键步骤包括：1. 通过领域特定提示词引导预训练大语言模型，为每个类别生成多样化的细粒度文本描述；2. 利用预训练视觉-语言模型为每张图像匹配保留标签且捕捉相关视觉特征的文本描述。我们通过将视觉-语言模型在图像-生成文本对上进行微调，学习对齐的视觉-语言表征空间以实现分类性能提升，从而验证了GIST的有效性。我们在四个不同领域的细粒度分类数据集上，分别在全样本和小样本场景下评估了所学表征空间。本方法在四个全样本数据集上相较于CLIP线性探测器平均提升4.1%的准确率，相较于先前最优的图文分类方法平均提升1.1%的准确率。在小样本场景下本方法也取得了类似改进。代码开源地址：https://github.com/emu1729/GIST。