Fine-grained visual classification (FGVC) involves categorizing fine subdivisions within a broader category, which poses challenges due to subtle inter-class discrepancies and large intra-class variations. However, prevailing approaches primarily focus on uni-modal visual concepts. Recent advancements in pre-trained vision-language models have demonstrated remarkable performance in various high-level vision tasks, yet the applicability of such models to FGVC tasks remains uncertain. In this paper, we aim to fully exploit the capabilities of cross-modal description to tackle FGVC tasks and propose a novel multimodal prompting solution, denoted as MP-FGVC, based on the contrastive language-image pertaining (CLIP) model. Our MP-FGVC comprises a multimodal prompts scheme and a multimodal adaptation scheme. The former includes Subcategory-specific Vision Prompt (SsVP) and Discrepancy-aware Text Prompt (DaTP), which explicitly highlights the subcategory-specific discrepancies from the perspectives of both vision and language. The latter aligns the vision and text prompting elements in a common semantic space, facilitating cross-modal collaborative reasoning through a Vision-Language Fusion Module (VLFM) for further improvement on FGVC. Moreover, we tailor a two-stage optimization strategy for MP-FGVC to fully leverage the pre-trained CLIP model and expedite efficient adaptation for FGVC. Extensive experiments conducted on four FGVC datasets demonstrate the effectiveness of our MP-FGVC.

翻译：细粒度视觉分类（FGVC）涉及对更广泛类别中的精细子类进行划分，由于类间细微差异和类内巨大变化而带来挑战。然而，现有方法主要关注单模态视觉概念。近期预训练的视觉-语言模型已在多种高层视觉任务中展现出卓越性能，但此类模型在FGVC任务中的适用性仍不明确。本文旨在充分利用跨模态描述的能力来解决FGVC任务，并提出一种基于对比语言-图像预训练（CLIP）模型的新型多模态提示解决方案——MP-FGVC。我们的MP-FGVC包含多模态提示方案和多模态自适应方案：前者包括子类别特定视觉提示（SsVP）和差异感知文本提示（DaTP），从视觉和语言两个角度明确突出子类别特定差异；后者将视觉和文本提示元素对齐到共同语义空间，通过视觉-语言融合模块（VLFM）促进跨模态协同推理，从而进一步改进FGVC。此外，我们为MP-FGVC定制了两阶段优化策略，以充分利用预训练的CLIP模型并加速FGVC的高效适配。在四个FGVC数据集上进行的广泛实验证明了我们的MP-FGVC的有效性。