Recent advances in instruction-tuned Large Vision-Language Models (LVLMs) have imbued the models with the ability to generate high-level, image-grounded explanations with ease. While such capability is largely attributed to the rich world knowledge contained within the Large Language Models (LLMs), our work reveals their shortcomings in fine-grained visual categorization (FGVC) across six different benchmark settings. Most recent state-of-the-art LVLMs like LLaVa-1.5, InstructBLIP and GPT-4V not only severely deteriorate in terms of classification performance, e.g., average drop of 65.58 in EM for Stanford Dogs for LLaVA-1.5, but also struggle to generate an accurate explanation with detailed attributes based on the concept that appears within an input image despite their capability to generate holistic image-level descriptions. In-depth analyses show that instruction-tuned LVLMs exhibit modality gap, showing discrepancy when given textual and visual inputs that correspond to the same concept, preventing the image modality from leveraging the rich parametric knowledge within the LLMs. In an effort to further the community's endeavor in this direction, we propose a multiple granularity attribute-centric evaluation benchmark, Finer, which aims to establish a ground to evaluate LVLMs' fine-grained visual comprehension ability and provide significantly improved explainability.

翻译：近期，经指令微调的大型视觉-语言模型（LVLMs）在生成基于图像的高层级描述方面展现出显著能力。尽管该能力主要归因于大型语言模型（LLMs）所蕴含的丰富世界知识，但我们的研究发现，这些模型在六个不同基准测试中的细粒度视觉分类（FGVC）任务上存在明显不足。当前最先进的LVLMs（如LLaVa-1.5、InstructBLIP和GPT-4V）不仅分类性能严重退化（例如，LLaVA-1.5在Stanford Dogs数据集上的精确匹配率平均下降65.58%），而且尽管具备生成整体图像描述的能力，却难以基于输入图像中的具体概念生成包含详细属性的准确解释。深入分析表明，经指令微调的LVLMs存在模态鸿沟现象：当文本与视觉输入对应同一概念时，模型对二者的表征存在差异，阻碍了图像模态充分利用LLMs中的丰富参数化知识。为推动该领域研究进展，我们提出了一个多粒度属性中心评估基准——FineR，旨在建立评估LVLMs细粒度视觉理解能力的基础框架，并提供显著增强的可解释性。