We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new capabilities of perceiving object descriptions (e.g., bounding boxes) and grounding text to the visual world. Specifically, we represent refer expressions as links in Markdown, i.e., ``[text span](bounding boxes)'', where object descriptions are sequences of location tokens. Together with multimodal corpora, we construct large-scale data of grounded image-text pairs (called GrIT) to train the model. In addition to the existing capabilities of MLLMs (e.g., perceiving general modalities, following instructions, and performing in-context learning), Kosmos-2 integrates the grounding capability into downstream applications. We evaluate Kosmos-2 on a wide range of tasks, including (i) multimodal grounding, such as referring expression comprehension, and phrase grounding, (ii) multimodal referring, such as referring expression generation, (iii) perception-language tasks, and (iv) language understanding and generation. This work lays out the foundation for the development of Embodiment AI and sheds light on the big convergence of language, multimodal perception, action, and world modeling, which is a key step toward artificial general intelligence. Data, demo, and pretrained models are available at https://aka.ms/kosmos-2.

翻译：我们提出Kosmos-2——一种多模态大语言模型（MLLM），实现了感知物体描述（如边界框）及将文本锚定至视觉世界的新能力。具体而言，我们将指代表达表示为Markdown形式的链接，即“[文本段](边界框)”，其中物体描述由位置标记序列构成。结合多模态语料库，我们构建了大规模锚定图像-文本对数据集（称为GrIT）以训练模型。除MLLM现有能力（如感知通用模态、遵循指令及执行上下文学习）之外，Kosmos-2将锚定能力集成至下游应用。我们在广泛任务上评估了Kosmos-2，包括：（i）多模态锚定，如指代表达理解与短语锚定；（ii）多模态指代，如指代表达生成；（iii）感知-语言任务；（iv）语言理解与生成。本工作为具身AI的发展奠定基础，并揭示了语言、多模态感知、动作与世界模型的大一统趋势——这将是迈向通用人工智能的关键一步。数据、演示及预训练模型已开源至https://aka.ms/kosmos-2。