A promising approach for improving the performance of vision-language models like CLIP for image classification is to extend the class descriptions (i.e., prompts) with related attributes, e.g., using brown sparrow instead of sparrow. However, current zero-shot methods select a subset of attributes regardless of commonalities between the target classes, potentially providing no useful information that would have helped to distinguish between them. For instance, they may use color instead of bill shape to distinguish between sparrows and wrens, which are both brown. We propose Follow-up Differential Descriptions (FuDD), a zero-shot approach that tailors the class descriptions to each dataset and leads to additional attributes that better differentiate the target classes. FuDD first identifies the ambiguous classes for each image, and then uses a Large Language Model (LLM) to generate new class descriptions that differentiate between them. The new class descriptions resolve the initial ambiguity and help predict the correct label. In our experiments, FuDD consistently outperforms generic description ensembles and naive LLM-generated descriptions on 12 datasets. We show that differential descriptions are an effective tool to resolve class ambiguities, which otherwise significantly degrade the performance. We also show that high quality natural language class descriptions produced by FuDD result in comparable performance to few-shot adaptation methods.

翻译：提升视觉-语言模型（如CLIP）在图像分类任务中的一种有效途径是扩展类别描述（即提示词），通过引入相关属性实现，例如用"棕色麻雀"替代"麻雀"。然而，现有零样本方法无论目标类别间的共性如何，均选择固定属性子集，可能导致无法提供区分性信息——例如在区分同为棕色的麻雀与鹪鹩时，可能使用颜色特征而非喙部形状特征。本文提出**跟进式差异描述**（FuDD），一种针对数据集定制类别描述的零样本方法，通过生成更具区分性的附加属性来提升分类效果。FuDD首先识别每个图像中的歧义类别，随后利用大语言模型（LLM）生成差异化类别描述以消除歧义。这些新生成的描述有效解决初始分类歧义，帮助预测正确标签。实验表明，在12个数据集上，FuDD始终优于通用描述集成及朴素LLM生成描述。我们证明差异化描述是解决类别歧义的有效工具——这些歧义本会显著降低模型性能。同时，FuDD产生的高质量自然语言类别描述性能可与少样本自适应方法相媲美。