Multi-modal large language models have demonstrated impressive performance across various tasks in different modalities. However, existing multi-modal models primarily emphasize capturing global information within each modality while neglecting the importance of perceiving local information across modalities. Consequently, these models lack the ability to effectively understand the fine-grained details of input data, limiting their performance in tasks that require a more nuanced understanding. To address this limitation, there is a compelling need to develop models that enable fine-grained understanding across multiple modalities, thereby enhancing their applicability to a wide range of tasks. In this paper, we propose LEGO, a language enhanced multi-modal grounding model. Beyond capturing global information like other multi-modal models, our proposed model excels at tasks demanding a detailed understanding of local information within the input. It demonstrates precise identification and localization of specific regions in images or moments in videos. To achieve this objective, we design a diversified dataset construction pipeline, resulting in a multi-modal, multi-granularity dataset for model training. The code, dataset, and demo of our model can be found at https: //github.com/lzw-lzw/LEGO.

翻译：多模态大语言模型在各类跨模态任务中展现出卓越性能。然而，现有多模态模型主要侧重于捕捉各模态内的全局信息，忽视了跨模态局部信息感知的重要性。这导致模型缺乏对输入数据细粒度细节的有效理解能力，制约了其在需要更精细化理解的任务中的表现。为解决这一局限，亟需开发能够实现跨模态细粒度理解的模型，从而增强其在广泛任务中的适用性。本文提出语言增强的多模态基础模型LEGO。与仅捕捉全局信息的其他多模态模型不同，本模型在处理需要输入局部细节理解的任务中表现优异，能精准识别并定位图像中的特定区域或视频中的特定时刻。为实现该目标，我们设计了多样化数据集构建流水线，生成了多模态多粒度训练数据集。模型代码、数据集及演示可访问https://github.com/lzw-lzw/LEGO获取。