Multi-modal Large Language Models (MLLMs) have shown remarkable capabilities in various multi-modal tasks. Nevertheless, their performance in fine-grained image understanding tasks is still limited. To address this issue, this paper proposes a new framework to enhance the fine-grained image understanding abilities of MLLMs. Specifically, we present a new method for constructing the instruction tuning dataset at a low cost by leveraging annotations in existing datasets. A self-consistent bootstrapping method is also introduced to extend existing dense object annotations into high-quality referring-expression-bounding-box pairs. These methods enable the generation of high-quality instruction data which includes a wide range of fundamental abilities essential for fine-grained image perception. Moreover, we argue that the visual encoder should be tuned during instruction tuning to mitigate the gap between full image perception and fine-grained image perception. Experimental results demonstrate the superior performance of our method. For instance, our model exhibits a 5.2% accuracy improvement over Qwen-VL on GQA and surpasses the accuracy of Kosmos-2 by 24.7% on RefCOCO_val. We also attain the top rank on the leaderboard of MMBench. This promising performance is achieved by training on only publicly available data, making it easily reproducible. The models, datasets, and codes are publicly available at https://github.com/SY-Xuan/Pink.

翻译：摘要：多模态大语言模型（MLLMs）在各类多模态任务中展现出卓越能力。然而，它们在细粒度图像理解任务中的表现仍存在局限。为解决这一问题，本文提出一种新框架以增强MLLMs的细粒度图像理解能力。具体而言，我们提出一种低成本构建指令微调数据集的新方法，通过利用现有数据集中的标注信息。同时引入一种自洽自举方法，将现有密集目标标注扩展为高质量的指代表达-边界框对。这些方法能够生成包含细粒度图像感知所需广泛基础能力的高质量指令数据。此外，我们认为在指令微调过程中应对视觉编码器进行微调，以弥合全局图像感知与细粒度图像感知之间的差距。实验结果表明本方法具有优越性能。例如，我们的模型在GQA上相比Qwen-VL准确率提升5.2%，在RefCOCO_val上超越Kosmos-2准确率24.7%，并在MMBench排行榜上取得首位。这一优异性能仅通过使用公开数据集训练实现，因此易于复现。模型、数据集及代码已开源发布于https://github.com/SY-Xuan/Pink。