We present Set-of-Mark (SoM), a new visual prompting method, to unleash the visual grounding abilities of large multimodal models (LMMs), such as GPT-4V. As illustrated in Fig. 1 (right), we employ off-the-shelf interactive segmentation models, such as SAM, to partition an image into regions at different levels of granularity, and overlay these regions with a set of marks e.g., alphanumerics, masks, boxes. Using the marked image as input, GPT-4V can answer the questions that require visual grounding. We perform a comprehensive empirical study to validate the effectiveness of SoM on a wide range of fine-grained vision and multimodal tasks. For example, our experiments show that GPT-4V with SoM outperforms the state-of-the-art fully-finetuned referring segmentation model on RefCOCOg in a zero-shot setting.

翻译：我们提出Set-of-Mark (SoM)，一种新型视觉提示方法，用于释放大型多模态模型（如GPT-4V）的视觉定位能力。如图1（右侧）所示，我们采用现成的交互式分割模型（如SAM）将图像划分为不同粒度级别的区域，并为这些区域叠加一组标记（例如字母数字符号、掩码、方框）。以标记后的图像作为输入，GPT-4V能够回答需要视觉定位的问题。我们通过全面的实证研究验证了SoM在多种细粒度视觉和多模态任务中的有效性。例如，实验表明，采用SoM的GPT-4V在零样本设置下，于RefCOCOg数据集上超越了当前最先进的完全微调指代分割模型。