Recent advances in instruction-tuned Large Vision-Language Models (LVLMs) have imbued the models with the ability to generate high-level, image-grounded explanations with ease. While such capability is largely attributed to the rich world knowledge contained within the Large Language Models (LLMs), our work reveals their shortcomings in fine-grained visual categorization (FGVC) across six different benchmark settings. Most recent state-of-the-art LVLMs like LLaVa-1.5, InstructBLIP and GPT-4V not only severely deteriorate in terms of classification performance, e.g., average drop of 65.58 in EM for Stanford Dogs for LLaVA-1.5, but also struggle to generate an accurate explanation with detailed attributes based on the concept that appears within an input image despite their capability to generate holistic image-level descriptions. In-depth analyses show that instruction-tuned LVLMs exhibit modality gap, showing discrepancy when given textual and visual inputs that correspond to the same concept, preventing the image modality from leveraging the rich parametric knowledge within the LLMs. In an effort to further the community's endeavor in this direction, we propose a multiple granularity attribute-centric evaluation benchmark, Finer, which aims to establish a ground to evaluate LVLMs' fine-grained visual comprehension ability and provide significantly improved explainability.

翻译：近期，经过指令微调的大型视觉语言模型（LVLMs）取得了进展，使其能够轻松生成基于图像的高层次解释。尽管这种能力在很大程度上归因于大型语言模型（LLMs）所蕴含的丰富世界知识，但我们的研究揭示了它们在六种不同基准测试中细粒度视觉分类（FGVC）方面的不足。最新的最先进LVLMs（如LLaVa-1.5、InstructBLIP和GPT-4V）不仅在分类性能上严重下降（例如，LLaVA-1.5在Stanford Dogs数据集上的精确匹配率平均下降65.58），而且尽管它们能够生成整体图像级描述，但在基于输入图像中出现的概念生成带有详细属性的准确解释时仍存在困难。深入分析表明，经过指令微调的LVLMs存在模态差距，即当文本和视觉输入对应同一概念时出现差异，阻碍了图像模态利用LLMs中的丰富参数化知识。为了推动该领域的研究进展，我们提出了一个多粒度属性为中心的评估基准Finer，旨在为评估LVLMs的细粒度视觉理解能力建立基础，并提供显著增强的可解释性。