The remarkable success of Large Language Models (LLMs) and instruction tuning drives the evolution of Vision Language Models (VLMs) towards a versatile general-purpose model. Yet, it remains unexplored whether current VLMs genuinely possess quality object-level image understanding capabilities determined from `what objects are in the image?' or `which object corresponds to a specified bounding box?'. Our findings reveal that the image understanding capabilities of current VLMs are strongly correlated with their zero-shot performance on vision language (VL) tasks. This suggests that prioritizing basic image understanding is crucial for VLMs to excel at VL tasks. To enhance object-level image understanding, we propose Crayon Large Language and Vision mOdel(CoLLaVO), which incorporates instruction tuning with Crayon Prompt as a new visual prompt tuning scheme based on panoptic color maps. Furthermore, we present a learning strategy of Dual QLoRA to preserve object-level image understanding without forgetting it during visual instruction tuning, thereby achieving a significant leap in numerous VL benchmarks in a zero-shot setting.

翻译：大语言模型（LLMs）和指令微调的巨大成功正推动视觉语言模型（VLMs）向通用模型演进。然而，当前VLMs是否真正具备由“图像中存在哪些物体？”或“哪个物体对应指定边界框？”所决定的高质量物体级图像理解能力，目前尚未得到充分探索。我们的研究发现，当前VLMs的图像理解能力与其在视觉语言（VL）任务上的零样本表现密切相关。这表明提升基础图像理解能力对于VLMs在VL任务中取得优异表现至关重要。为增强物体级图像理解，我们提出CoLLaVO（Crayon Large Language and Vision mOdel），该模型引入了基于全景颜色图的蜡笔提示（Crayon Prompt）作为新型视觉提示微调方案，并配合指令微调。此外，我们提出了双QLoRA学习策略，在视觉指令微调过程中保持物体级图像理解能力不被遗忘，从而在众多VL基准测试的零样本场景下实现显著性能飞跃。