In human conversations, individuals can indicate relevant regions within a scene while addressing others. In turn, the other person can then respond by referring to specific regions if necessary. This natural referential ability in dialogue remains absent in current Multimodal Large Language Models (MLLMs). To fill this gap, this paper proposes an MLLM called Shikra, which can handle spatial coordinate inputs and outputs in natural language. Its architecture consists of a vision encoder, an alignment layer, and a LLM. It is designed to be straightforward and simple, without the need for extra vocabularies, position encoder, pre-/post-detection modules, or external plug-in models. All inputs and outputs are in natural language form. Referential dialogue is a superset of various vision-language (VL) tasks. Shikra can naturally handle location-related tasks like REC and PointQA, as well as conventional VL tasks such as Image Captioning and VQA. Experimental results showcase Shikra's promising performance. Furthermore, it enables numerous exciting applications, like providing mentioned objects' coordinates in chains of thoughts and comparing user-pointed regions similarities. Our code, model and dataset are accessed at https://github.com/shikras/shikra.

翻译：在人类对话中，个体可以一边指向他人，一边指出场景中的相关区域。而另一方则能在必要时通过指代特定区域做出回应。这种自然的指代能力在当前的多模态大语言模型中尚属空白。为填补这一空缺，本文提出名为Shikra的多模态大语言模型，该模型能够处理自然语言形式的空间坐标输入与输出。其架构由视觉编码器、对齐层和大语言模型组成，设计简洁直接，无需额外词汇表、位置编码器、前/后检测模块或外部插件模型。所有输入与输出均采用自然语言形式。指代对话是多种视觉-语言任务的超集。Shikra既能自然处理REC和PointQA等定位相关任务，也能胜任图像描述和视觉问答等传统视觉-语言任务。实验结果表明Shikra具有令人期待的性能表现。此外，该模型还支持多种激动人心的应用场景，例如在思维链中提供提及对象的坐标，以及比较用户所指区域间的相似度。我们的代码、模型及数据集可通过https://github.com/shikras/shikra获取。