Existing instance segmentation models learn task-specific information using manual mask annotations from base (training) categories. These mask annotations require tremendous human effort, limiting the scalability to annotate novel (new) categories. To alleviate this problem, Open-Vocabulary (OV) methods leverage large-scale image-caption pairs and vision-language models to learn novel categories. In summary, an OV method learns task-specific information using strong supervision from base annotations and novel category information using weak supervision from image-captions pairs. This difference between strong and weak supervision leads to overfitting on base categories, resulting in poor generalization towards novel categories. In this work, we overcome this issue by learning both base and novel categories from pseudo-mask annotations generated by the vision-language model in a weakly supervised manner using our proposed Mask-free OVIS pipeline. Our method automatically generates pseudo-mask annotations by leveraging the localization ability of a pre-trained vision-language model for objects present in image-caption pairs. The generated pseudo-mask annotations are then used to supervise an instance segmentation model, freeing the entire pipeline from any labour-expensive instance-level annotations and overfitting. Our extensive experiments show that our method trained with just pseudo-masks significantly improves the mAP scores on the MS-COCO dataset and OpenImages dataset compared to the recent state-of-the-art methods trained with manual masks. Codes and models are provided in https://vibashan.github.io/ovis-web/.

翻译：现有实例分割模型利用基础（训练）类别的掩膜标注学习任务特定信息。这些掩膜标注需要大量人力，限制了向新类别的标注可扩展性。为解决该问题，开放词汇（OV）方法利用大规模图像-文本对和视觉-语言模型学习新类别。简言之，OV方法通过基础标注的强监督信息学习任务特定知识，并借助图像-文本对的弱监督信息学习新类别知识。这种强监督与弱监督的差异导致模型过拟合于基础类别，从而对新类别的泛化能力较差。本文通过提出的无掩膜OVIS流程，以弱监督方式利用视觉-语言模型生成的伪掩膜标注同时学习基础类别与新类别，克服了上述问题。本方法利用预训练视觉-语言模型对图像-文本对中目标的定位能力自动生成伪掩膜标注。生成的伪掩膜标注随后用于监督实例分割模型，使整个流程摆脱任何繁重的实例级标注和过拟合问题。大量实验表明，仅使用伪掩膜训练的本方法在MS-COCO数据集和OpenImages数据集上的mAP分数显著超越了近期使用人工掩膜训练的最先进方法。代码与模型已在 https://vibashan.github.io/ovis-web/ 公开。