We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new capabilities of perceiving object descriptions (e.g., bounding boxes) and grounding text to the visual world. Specifically, we represent refer expressions as links in Markdown, i.e., ``[text span](bounding boxes)'', where object descriptions are sequences of location tokens. Together with multimodal corpora, we construct large-scale data of grounded image-text pairs (called GrIT) to train the model. In addition to the existing capabilities of MLLMs (e.g., perceiving general modalities, following instructions, and performing in-context learning), Kosmos-2 integrates the grounding capability into downstream applications. We evaluate Kosmos-2 on a wide range of tasks, including (i) multimodal grounding, such as referring expression comprehension, and phrase grounding, (ii) multimodal referring, such as referring expression generation, (iii) perception-language tasks, and (iv) language understanding and generation. This work lays out the foundation for the development of Embodiment AI and sheds light on the big convergence of language, multimodal perception, action, and world modeling, which is a key step toward artificial general intelligence. Code and pretrained models are available at https://aka.ms/kosmos-2.

翻译：我们提出了Kosmos-2，一种多模态大语言模型（MLLM），它赋予了模型感知物体描述（如边界框）以及将文本锚定至视觉世界的新能力。具体而言，我们使用Markdown格式中的链接来表示引用表达式，即“[文本片段](边界框)”，其中物体描述由一系列位置标记组成。结合多模态语料，我们构建了大规模锚定图像-文本对数据集（称为GrIT）用于训练该模型。除MLLM现有能力（如感知通用模态、遵循指令以及执行上下文学习）之外，Kosmos-2还将锚定能力集成至下游应用中。我们在广泛的任务上评估了Kosmos-2，包括：（i）多模态锚定，如引用表达式理解和短语锚定；（ii）多模态引用，如引用表达式生成；（iii）感知-语言任务；以及（iv）语言理解与生成。本工作为具身智能的发展奠定了基础，并揭示了语言、多模态感知、动作与世界模型间的大融合，这是迈向通用人工智能的关键一步。代码与预训练模型已发布于 https://aka.ms/kosmos-2。