Despite the impressive capabilities of Multimodal Large Language Models (MLLMs) in integrating text and image modalities, challenges remain in accurately interpreting detailed visual elements. This paper presents an empirical study on enhancing MLLMs with state-of-the-art (SOTA) object detection and Optical Character Recognition (OCR) models to improve fine-grained understanding and reduce hallucination in responses. We investigate the embedding-based infusion of textual detection information, the impact of such infusion on MLLMs' original abilities, and the interchangeability of detection models. We conduct systematic and extensive experiments with representative models such as LLaVA-1.5, DINO, PaddleOCRv2, and Grounding DINO, revealing that our simple yet general approach not only refines MLLMs' performance in fine-grained visual tasks but also maintains their original strengths. Notably, the enhanced LLaVA-1.5 outperforms its original 7B/13B models on all 10 benchmarks, achieving an improvement of up to 12.5% on the normalized average score. We release our codes to facilitate further exploration into the fine-grained multimodal capabilities of MLLMs.

翻译：尽管多模态大语言模型（MLLMs）在整合文本与图像模态方面展现出卓越能力，其在精确解析细粒度视觉元素方面仍面临挑战。本文通过实证研究，探讨如何利用最先进的物体检测与光学字符识别（OCR）模型增强MLLMs，以提升其细粒度理解能力并减少回答中的幻觉现象。我们研究了基于嵌入的文本检测信息注入方法、此类注入对MLLMs原有能力的影响，以及检测模型的可互换性。通过对LLaVA-1.5、DINO、PaddleOCRv2和Grounding DINO等代表性模型进行系统化大规模实验，我们发现这种简洁通用的方法不仅能优化MLLMs在细粒度视觉任务中的表现，还能保持其原有优势。值得注意的是，增强版LLaVA-1.5在全部10个基准测试中均超越原始7B/13B模型，标准化平均分数最高提升达12.5%。我们已公开相关代码，以促进对MLLMs细粒度多模态能力的进一步探索。