In this study, we establish a baseline for a new task named multimodal multi-round referring and grounding (MRG), opening up a promising direction for instance-level multimodal dialogues. We present a new benchmark and an efficient vision-language model for this purpose. The new benchmark, named CB-300K, spans challenges including multi-round dialogue, complex spatial relationships among multiple instances, and consistent reasoning, which are beyond those shown in existing benchmarks. The proposed model, named ChatterBox, utilizes a two-branch architecture to collaboratively handle vision and language tasks. By tokenizing instance regions, the language branch acquires the ability to perceive referential information. Meanwhile, ChatterBox feeds a query embedding in the vision branch to a token receiver for visual grounding. A two-stage optimization strategy is devised, making use of both CB-300K and auxiliary external data to improve the model's stability and capacity for instance-level understanding. Experiments show that ChatterBox outperforms existing models in MRG both quantitatively and qualitatively, paving a new path towards multimodal dialogue scenarios with complicated and precise interactions. Code, data, and model are available at: https://github.com/sunsmarterjie/ChatterBox.

翻译：在本研究中，我们为一项名为多模态多轮指代与定位（MRG）的新任务建立了基线，开辟了实例级多模态对话的有前景方向。为此，我们提出了一个新的基准和一个高效的视觉-语言模型。新基准名为CB-300K，涵盖了多轮对话、多实例间的复杂空间关系以及一致性推理等挑战，超越了现有基准中的设定。所提出的模型名为ChatterBox，采用双分支架构协同处理视觉与语言任务。通过对实例区域进行标记化，语言分支获得了感知指代信息的能力。同时，ChatterBox在视觉分支中向标记接收器输入查询嵌入以实现视觉定位。我们设计了两阶段优化策略，利用CB-300K及辅助外部数据提升模型的稳定性与实例级理解能力。实验表明，ChatterBox在MRG任务上无论定量还是定性表现均优于现有模型，为复杂精确交互的多模态对话场景开辟了新路径。代码、数据和模型详见：https://github.com/sunsmarterjie/ChatterBox。