Despite the growing demand for interactive AI systems, there have been few comprehensive studies on human-AI interaction in visual understanding e.g. segmentation. Inspired by the development of prompt-based universal interfaces for LLMs, this paper presents SEEM, a promptable, interactive model for Segmenting Everything Everywhere all at once in an image. SEEM has four desiderata: i) Versatility: by introducing a versatile prompting engine for different types of prompts, including points, boxes, scribbles, masks, texts, and referred regions of another image; ii) Compositionality: by learning a joint visual-semantic space for visual and textual prompts to compose queries on the fly for inference as shown in Fig 1; iii)Interactivity: by incorporating learnable memory prompts to retain dialog history information via mask-guided cross-attention; and iv) Semantic-awareness: by using a text encoder to encode text queries and mask labels for open-vocabulary segmentation.

翻译：尽管交互式AI系统的需求日益增长，但在视觉理解（例如分割）领域，关于人机交互的研究仍相对匮乏。受大语言模型中基于提示的通用界面发展启发，本文提出SEEM——一种可提示的交互式模型，能够在一幅图像中同时分割一切。SEEM具备四个关键特性：i) 多功能性：通过引入通用提示引擎，支持多种提示类型，包括点、框、涂鸦、掩码、文本以及另一图像中的参考区域；ii) 组合性：通过学习联合视觉-语义空间，将视觉与文本提示组合为推理时的即时查询（如图1所示）；iii) 交互性：通过引入可学习的记忆提示，并利用掩码引导的交叉注意力机制保留对话历史信息；iv) 语义感知：利用文本编码器对文本查询和掩码标签进行编码，实现开放词汇分割。