Multi-modal large language models have demonstrated impressive performance across various tasks in different modalities. However, existing multi-modal models primarily emphasize capturing global information within each modality while neglecting the importance of perceiving local information across modalities. Consequently, these models lack the ability to effectively understand the fine-grained details of input data, limiting their performance in tasks that require a more nuanced understanding. To address this limitation, there is a compelling need to develop models that enable fine-grained understanding across multiple modalities, thereby enhancing their applicability to a wide range of tasks. In this paper, we propose GroundingGPT, a language enhanced multi-modal grounding model. Beyond capturing global information like other multi-modal models, our proposed model excels at tasks demanding a detailed understanding of local information within the input. It demonstrates precise identification and localization of specific regions in images or moments in videos. To achieve this objective, we design a diversified dataset construction pipeline, resulting in a multi-modal, multi-granularity dataset for model training. The code, dataset, and demo of our model can be found at https: //github.com/lzw-lzw/GroundingGPT.

翻译：多模态大语言模型在多种模态的不同任务中展示了令人瞩目的性能。然而，现有多模态模型主要侧重于捕捉各模态内的全局信息，忽视了跨模态局部信息感知的重要性，导致其难以有效理解输入数据的细粒度细节，限制了在需要更细腻理解的任务中的表现。为解决这一局限，亟需开发能够实现跨模态细粒度理解的模型，从而提升其在广泛任务中的适用性。本文提出GroundingGPT，一种语言增强的多模态定位模型。该模型不仅能像其他多模态模型一样捕捉全局信息，在需要细致理解输入局部信息的任务中表现尤为出色，能够精准识别并定位图像中的特定区域或视频中的特定时刻。为实现这一目标，我们设计了一种多样化的数据集构建流程，生成了用于模型训练的多模态、多粒度数据集。模型的代码、数据集和演示可访问网址：https: //github.com/lzw-lzw/GroundingGPT。