In this paper, we consider the problem of open-vocabulary semantic segmentation (OVS), which aims to segment objects of arbitrary classes instead of pre-defined, closed-set categories. The main contributions are as follows: First, we propose a transformer-based model for OVS, termed as OVSegmentor, which only exploits web-crawled image-text pairs for pre-training without using any mask annotations. OVSegmentor assembles the image pixels into a set of learnable group tokens via a slot-attention based binding module, and aligns the group tokens to the corresponding caption embedding. Second, we propose two proxy tasks for training, namely masked entity completion and cross-image mask consistency. The former aims to infer all masked entities in the caption given the group tokens, that enables the model to learn fine-grained alignment between visual groups and text entities. The latter enforces consistent mask predictions between images that contain shared entities, which encourages the model to learn visual invariance. Third, we construct CC4M dataset for pre-training by filtering CC12M with frequently appeared entities, which significantly improves training efficiency. Fourth, we perform zero-shot transfer on three benchmark datasets, PASCAL VOC 2012, PASCAL Context, and COCO Object. Our model achieves superior segmentation results over the state-of-the-art method by using only 3\% data (4M vs 134M) for pre-training. Code and pre-trained models will be released for future research.

翻译：本文研究了开放词汇语义分割（OVS）问题，旨在对任意类别的物体进行分割，而非预定义的封闭集类别。主要贡献如下：首先，我们提出了一种基于Transformer的OVS模型，称为OVSegmentor，该模型仅利用网络爬取的图像-文本对进行预训练，无需任何掩码标注。OVSegmentor通过基于槽注意力（slot-attention）的绑定模块将图像像素聚合成一组可学习的组令牌（group token），并将这些组令牌与对应的字幕嵌入对齐。其次，我们提出了两种训练代理任务，即掩码实体完成和跨图像掩码一致性。前者旨在根据组令牌推断字幕中所有被掩码的实体，使模型能够学习视觉组与文本实体之间的细粒度对齐；后者则对包含共享实体的图像强制执行一致的掩码预测，促使模型学习视觉不变性。第三，我们通过从CC12M中筛选高频出现实体构建了用于预训练的CC4M数据集，显著提升了训练效率。第四，我们在三个基准数据集（PASCAL VOC 2012、PASCAL Context和COCO Object）上进行了零样本迁移测试。我们的模型在仅使用3%数据（4M vs 134M）进行预训练的情况下，取得了优于现有最先进方法的分割结果。代码和预训练模型将发布以供未来研究使用。