Fine-grained visual classification (FGVC) involves categorizing fine subdivisions within a broader category, which poses challenges due to subtle inter-class discrepancies and large intra-class variations. However, prevailing approaches primarily focus on uni-modal visual concepts. Recent advancements in pre-trained vision-language models have demonstrated remarkable performance in various high-level vision tasks, yet the applicability of such models to FGVC tasks remains uncertain. In this paper, we aim to fully exploit the capabilities of cross-modal description to tackle FGVC tasks and propose a novel multimodal prompting solution, denoted as MP-FGVC, based on the contrastive language-image pertaining (CLIP) model. Our MP-FGVC comprises a multimodal prompts scheme and a multimodal adaptation scheme. The former includes Subcategory-specific Vision Prompt (SsVP) and Discrepancy-aware Text Prompt (DaTP), which explicitly highlights the subcategory-specific discrepancies from the perspectives of both vision and language. The latter aligns the vision and text prompting elements in a common semantic space, facilitating cross-modal collaborative reasoning through a Vision-Language Fusion Module (VLFM) for further improvement on FGVC. Moreover, we tailor a two-stage optimization strategy for MP-FGVC to fully leverage the pre-trained CLIP model and expedite efficient adaptation for FGVC. Extensive experiments conducted on four FGVC datasets demonstrate the effectiveness of our MP-FGVC.

翻译：细粒度视觉分类（FGVC）旨在对更广泛类别中的精细子类进行划分，由于类间差异细微且类内差异显著，这一任务具有挑战性。然而，现有方法主要关注单模态的视觉概念。近期预训练的视觉-语言模型在多种高级视觉任务中展现出卓越性能，但这些模型在FGVC任务中的适用性尚不明确。本文旨在充分利用跨模态描述的能力来解决FGVC任务，并提出一种基于对比语言-图像预训练（CLIP）模型的新型多模态提示解决方案，记为MP-FGVC。我们的MP-FGVC包含一个多模态提示方案和一个多模态适配方案。前者包括子类别特定视觉提示（SsVP）和差异感知文本提示（DaTP），分别从视觉和语言视角明确突出子类别特定的差异；后者将视觉和文本提示元素对齐到共同的语义空间，并通过视觉-语言融合模块（VLFM）促进跨模态协同推理，从而进一步提升FGVC性能。此外，我们为MP-FGVC定制了一个两阶段优化策略，以充分利用预训练的CLIP模型并加速对FGVC的高效适配。在四个FGVC数据集上进行的广泛实验证明了我们MP-FGVC的有效性。