The development of large language models (LLMs) has greatly advanced the field of multimodal understanding, leading to the emergence of large multimodal models (LMMs). In order to enhance the level of visual comprehension, recent studies have equipped LMMs with region-level understanding capabilities by representing object bounding box coordinates as a series of text sequences (pixel2seq). In this paper, we introduce a novel paradigm for object location modeling called pixel2emb method, where we ask the LMM to output the location embeddings and then decoded by different decoders. This paradigm allows for different location formats (such as bounding boxes and masks) to be used in multimodal conversations Furthermore, this kind of embedding based location modeling enables the utilization of existing practices in localization tasks, such as detection and segmentation. In scenarios with limited resources, our pixel2emb demonstrates superior performance compared to existing state-of-the-art (SOTA) approaches in both the location input and output tasks under fair comparison. Leveraging the proposed pixel2emb method, we train an LMM named NExT-Chat and demonstrate its capability of handling multiple tasks like visual grounding, region caption, and grounded reasoning.

翻译：大语言模型（LLMs）的发展极大地推动了多模态理解领域的进步，催生了大型多模态模型（LMMs）的出现。为提升视觉理解水平，近期研究通过将目标边界框坐标表示为文本序列（pixel2seq）的方法，赋予LMMs区域级理解能力。本文提出了一种名为pixel2emb的目标位置建模新范式，通过让LMM输出位置嵌入，再由不同解码器进行解码。该范式支持在多模态对话中使用不同位置格式（如边界框和掩码）。此外，这种基于嵌入的位置建模能够利用现有定位任务（如检测与分割）的成熟方法。在资源受限场景下，我们的pixel2emb方法在公平比较中，于位置输入与输出任务上均展现出优于现有最先进（SOTA）方法的性能。基于所提出的pixel2emb方法，我们训练了名为NExT-Chat的LMM，并展示了其在视觉定位、区域描述及基于区间的推理等多任务处理中的能力。