In this paper, we introduce an open-vocabulary panoptic segmentation model that effectively unifies the strengths of the Segment Anything Model (SAM) with the vision-language CLIP model in an end-to-end framework. While SAM excels in generating spatially-aware masks, it's decoder falls short in recognizing object class information and tends to oversegment without additional guidance. Existing approaches address this limitation by using multi-stage techniques and employing separate models to generate class-aware prompts, such as bounding boxes or segmentation masks. Our proposed method, PosSAM is an end-to-end model which leverages SAM's spatially rich features to produce instance-aware masks and harnesses CLIP's semantically discriminative features for effective instance classification. Specifically, we address the limitations of SAM and propose a novel Local Discriminative Pooling (LDP) module leveraging class-agnostic SAM and class-aware CLIP features for unbiased open-vocabulary classification. Furthermore, we introduce a Mask-Aware Selective Ensembling (MASE) algorithm that adaptively enhances the quality of generated masks and boosts the performance of open-vocabulary classification during inference for each image. We conducted extensive experiments to demonstrate our methods strong generalization properties across multiple datasets, achieving state-of-the-art performance with substantial improvements over SOTA open-vocabulary panoptic segmentation methods. In both COCO to ADE20K and ADE20K to COCO settings, PosSAM outperforms the previous state-of-the-art methods by a large margin, 2.4 PQ and 4.6 PQ, respectively. Project Website: https://vibashan.github.io/possam-web/.

翻译：在本文中，我们提出了一种开放词汇全景分割模型，该模型在端到端框架中有效融合了分割任意模型（SAM）与视觉语言CLIP模型的优势。SAM擅长生成具有空间感知能力的掩码，但其解码器无法有效识别目标类别信息，且在没有额外引导时倾向于过度分割。现有方法通过采用多阶段技术，并利用独立模型生成类别感知提示（如边界框或分割掩码）来缓解此问题。我们的方法PosSAM是一种端到端模型，它利用SAM的空间丰富特征生成实例感知掩码，并借助CLIP的语义判别特征实现有效的实例分类。具体而言，我们针对SAM的局限性，提出了一种新颖的局部判别池化（LDP）模块，该模块利用类别无关的SAM特征与类别感知的CLIP特征，实现无偏的开放词汇分类。此外，我们引入了一种掩码感知选择性集成（MASE）算法，该算法可在推理过程中针对每张图像自适应地提升生成掩码的质量，并增强开放词汇分类性能。通过大量实验，我们证明了该方法在多个数据集上均具备强大的泛化能力，在开放词汇全景分割方法中取得了最优性能，并实现了显著提升。在COCO到ADE20K以及ADE20K到COCO两种设置下，PosSAM分别以2.4 PQ和4.6 PQ的显著优势超越了先前的最优方法。项目网站：https://vibashan.github.io/possam-web/。