Multi-modal large language models have demonstrated impressive performance across various tasks in different modalities. However, existing multi-modal models primarily emphasize capturing global information within each modality while neglecting the importance of perceiving local information across modalities. Consequently, these models lack the ability to effectively understand the fine-grained details of input data, limiting their performance in tasks that require a more nuanced understanding. To address this limitation, there is a compelling need to develop models that enable fine-grained understanding across multiple modalities, thereby enhancing their applicability to a wide range of tasks. In this paper, we propose GroundingGPT, a language enhanced multi-modal grounding model. Beyond capturing global information like other multi-modal models, our proposed model excels at tasks demanding a detailed understanding of local information within the input. It demonstrates precise identification and localization of specific regions in images or moments in videos. To achieve this objective, we design a diversified dataset construction pipeline, resulting in a multi-modal, multi-granularity dataset for model training. The code, dataset, and demo of our model can be found at https: //github.com/lzw-lzw/GroundingGPT.

翻译：多模态大语言模型已在不同模态的多种任务中展现出令人瞩目的性能。然而，现有多模态模型主要侧重于捕捉各模态内的全局信息，却忽视了感知跨模态局部信息的重要性。因此，这些模型缺乏有效理解输入数据细粒度细节的能力，这限制了它们在需要更细致理解的任务中的表现。为解决这一局限，迫切需要开发能够实现跨模态细粒度理解的模型，从而增强其在广泛任务中的适用性。本文提出GroundingGPT，一种语言增强的多模态定位模型。与其他多模态模型一样，该模型不仅能捕捉全局信息，更在需要详细理解输入局部信息的任务中表现出色，能够精准识别并定位图像中的特定区域或视频中的特定时刻。为实现这一目标，我们设计了一套多样化的数据集构建流程，生成了一个用于模型训练的多模态、多粒度数据集。本模型的代码、数据集及演示可在https://github.com/lzw-lzw/GroundingGPT获取。