We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new capabilities of perceiving object descriptions (e.g., bounding boxes) and grounding text to the visual world. Specifically, we represent refer expressions as links in Markdown, i.e., ``[text span](bounding boxes)'', where object descriptions are sequences of location tokens. Together with multimodal corpora, we construct large-scale data of grounded image-text pairs (called GrIT) to train the model. In addition to the existing capabilities of MLLMs (e.g., perceiving general modalities, following instructions, and performing in-context learning), Kosmos-2 integrates the grounding capability into downstream applications. We evaluate Kosmos-2 on a wide range of tasks, including (i) multimodal grounding, such as referring expression comprehension, and phrase grounding, (ii) multimodal referring, such as referring expression generation, (iii) perception-language tasks, and (iv) language understanding and generation. This work lays out the foundation for the development of Embodiment AI and sheds light on the big convergence of language, multimodal perception, action, and world modeling, which is a key step toward artificial general intelligence. Data, demo, and pretrained models are available at https://aka.ms/kosmos-2.

翻译：我们提出Kosmos-2，一种多模态大语言模型（MLLM），它具备了感知物体描述（如边界框）以及将文本锚定至视觉世界的新能力。具体而言，我们采用Markdown格式将指代表达表示为链接，即"[文本片段](边界框)"，其中物体描述由一系列位置标记构成。通过结合多模态语料库，我们构建了大规模锚定图像-文本对数据（称为GrIT）来训练该模型。除现有MLLM能力（如感知通用模态、遵循指令及执行上下文学习）外，Kosmos-2将锚定能力整合至下游应用中。我们在广泛任务上评估Kosmos-2，包括：(i)多模态锚定（如指代表达理解与短语锚定），(ii)多模态指代（如指代表达生成），(iii)感知-语言任务，以及(iv)语言理解与生成。该工作为具身智能的发展奠定基础，并揭示了语言、多模态感知、行为与世界模型大融合的前景，这将是迈向通用人工智能的关键一步。数据、演示及预训练模型可从https://aka.ms/kosmos-2获取。