The development of large language models (LLMs) has greatly advanced the field of multimodal understanding, leading to the emergence of large multimodal models (LMMs). In order to enhance the level of visual comprehension, recent studies have equipped LMMs with region-level understanding capabilities by representing object bounding box coordinates as a series of text sequences (pixel2seq). In this paper, we introduce a novel paradigm for object location modeling called pixel2emb method, where we ask the LMM to output the location embeddings and then decoded by different decoders. This paradigm allows for different location formats (such as bounding boxes and masks) to be used in multimodal conversations Furthermore, this kind of embedding based location modeling enables the utilization of existing practices in localization tasks, such as detection and segmentation. In scenarios with limited resources, our pixel2emb demonstrates superior performance compared to existing state-of-the-art (SOTA) approaches in both the location input and output tasks under fair comparison. Leveraging the proposed pixel2emb method, we train an LMM named NExT-Chat and demonstrate its capability of handling multiple tasks like visual grounding, region caption, and grounded reasoning.

翻译：大型语言模型（LLMs）的进展极大地推动了多模态理解领域的发展，催生了大型多模态模型（LMMs）。为提升视觉理解水平，近期研究通过将目标边界框坐标表示为文本序列（pixel2seq），赋予LMMs区域级理解能力。本文提出一种新颖的目标位置建模范式——pixel2emb方法，该方法使LMM输出位置嵌入，再由不同解码器进行解码。该范式允许在多模态对话中灵活使用不同位置格式（如边界框和掩码）。此外，这种基于嵌入的位置建模能够充分利用检测与分割等定位任务的现有实践。在资源受限场景下，公平对比表明，我们的pixel2emb方法在位置输入与输出任务中均优于现有最先进（SOTA）方法。基于所提出的pixel2emb方法，我们训练了名为NExT-Chat的LMM，并验证了其在视觉定位、区域描述与基于定位的推理等多任务中的处理能力。