In this paper, we introduce Semantic-SAM, a universal image segmentation model to enable segment and recognize anything at any desired granularity. Our model offers two key advantages: semantic-awareness and granularity-abundance. To achieve semantic-awareness, we consolidate multiple datasets across three granularities and introduce decoupled classification for objects and parts. This allows our model to capture rich semantic information. For the multi-granularity capability, we propose a multi-choice learning scheme during training, enabling each click to generate masks at multiple levels that correspond to multiple ground-truth masks. Notably, this work represents the first attempt to jointly train a model on SA-1B, generic, and part segmentation datasets. Experimental results and visualizations demonstrate that our model successfully achieves semantic-awareness and granularity-abundance. Furthermore, combining SA-1B training with other segmentation tasks, such as panoptic and part segmentation, leads to performance improvements. We will provide code and a demo for further exploration and evaluation.

翻译：本文提出语义-SAM（Semantic-SAM），一种通用图像分割模型，可在任意期望粒度下实现一切物体的分割与识别。该模型具备两大核心优势：语义感知性与粒度丰富性。为获得语义感知能力，我们整合了三种不同粒度的多个数据集，并引入针对物体与部件的解耦分类策略，使模型能够捕获丰富的语义信息。针对多粒度能力，我们提出训练阶段的多选学习方案，支持单个点击生成对应多个真值掩码的多层级掩码。值得注意的是，本工作首次尝试联合训练SA-1B、通用分割与部件分割数据集。实验结果与可视化表明，该模型成功实现了语义感知性与粒度丰富性。此外，将SA-1B训练与其他分割任务（如全景分割与部件分割）相结合，可提升模型性能。我们将提供代码与演示以便进一步探索与评估。