Existing open-vocabulary image segmentation methods require a fine-tuning step on mask annotations and/or image-text datasets. Mask labels are labor-intensive, which limits the number of categories in segmentation datasets. As a result, the open-vocabulary capacity of pre-trained VLMs is severely reduced after fine-tuning. However, without fine-tuning, VLMs trained under weak image-text supervision tend to make suboptimal mask predictions when there are text queries referring to non-existing concepts in the image. To alleviate these issues, we introduce a novel recurrent framework that progressively filters out irrelevant texts and enhances mask quality without training efforts. The recurrent unit is a two-stage segmenter built upon a VLM with frozen weights. Thus, our model retains the VLM's broad vocabulary space and strengthens its segmentation capability. Experimental results show that our method outperforms not only the training-free counterparts, but also those fine-tuned with millions of additional data samples, and sets new state-of-the-art records for both zero-shot semantic and referring image segmentation tasks. Specifically, we improve the current record by 28.8, 16.0, and 6.9 mIoU on Pascal VOC, COCO Object, and Pascal Context.

翻译：现有开放词汇图像分割方法需要在掩码标注和/或图像-文本数据集上进行微调。掩码标签需要大量人工标注，限制了分割数据集的类别数量。因此，预训练视觉语言模型（VLM）在微调后其开放词汇能力严重下降。然而，若不进行微调，在弱图像-文本监督下训练的VLM在文本查询指向图像中不存在概念时，往往会生成次优的掩码预测。为解决这些问题，我们提出了一种新颖的循环框架，该框架能够渐进式过滤无关文本并提升掩码质量，且无需任何训练过程。循环单元是一个基于冻结权重VLM构建的两阶段分割器。因此，我们的模型保留了VLM广泛的词汇空间并增强了其分割能力。实验结果表明，我们的方法不仅超越了无需训练的同类方法，还超越了那些使用数百万额外数据样本进行微调的方法，并在零样本语义分割和指代图像分割任务上均创下新纪录。具体而言，我们在Pascal VOC、COCO Object和Pascal Context数据集上分别将当前最佳结果提升了28.8、16.0和6.9个mIoU。