Recent vision-language models outperform vision-only models on many image classification tasks. However, because of the absence of paired text/image descriptions, it remains difficult to fine-tune these models for fine-grained image classification. In this work, we propose a method, GIST, for generating image-specific fine-grained text descriptions from image-only datasets, and show that these text descriptions can be used to improve classification. Key parts of our method include 1. prompting a pretrained large language model with domain-specific prompts to generate diverse fine-grained text descriptions for each class and 2. using a pretrained vision-language model to match each image to label-preserving text descriptions that capture relevant visual features in the image. We demonstrate the utility of GIST by fine-tuning vision-language models on the image-and-generated-text pairs to learn an aligned vision-language representation space for improved classification. We evaluate our learned representation space in full-shot and few-shot scenarios across four diverse fine-grained classification datasets, each from a different domain. Our method achieves an average improvement of $4.1\%$ in accuracy over CLIP linear probes and an average of $1.1\%$ improvement in accuracy over the previous state-of-the-art image-text classification method on the full-shot datasets. Our method achieves similar improvements across few-shot regimes. Code is available at https://github.com/emu1729/GIST.

翻译：近期视觉-语言模型在诸多图像分类任务上超越了纯视觉模型。然而，由于缺乏配对的文本/图像描述，微调这些模型以进行细粒度图像分类仍具挑战性。本文提出GIST方法，从仅含图像的数据集中生成图像专属的细粒度文本描述，并证明这些文本描述可用于改进分类。方法核心包括：1）利用领域特定提示词引导预训练大语言模型为每个类别生成多样化的细粒度文本描述；2）使用预训练视觉-语言模型为每张图像匹配保留标签的文本描述，以捕捉图像中的相关视觉特征。我们通过将视觉-语言模型在图像-生成文本对上进行微调，学习对齐的视觉-语言表征空间以提升分类性能，从而验证GIST的有效性。在四个不同领域的细粒度分类数据集上，我们分别在全样本和少样本场景中评估了所学表征空间。与CLIP线性探测相比，本方法在全样本数据集上平均准确率提升4.1%；与先前最优的图像-文本分类方法相比，平均准确率提升1.1%。在少样本场景中同样获得相近提升。代码开源于https://github.com/emu1729/GIST。