Existing open-vocabulary image segmentation methods require a fine-tuning step on mask annotations and/or image-text datasets. Mask labels are labor-intensive, which limits the number of categories in segmentation datasets. As a result, the open-vocabulary capacity of pre-trained VLMs is severely reduced after fine-tuning. However, without fine-tuning, VLMs trained under weak image-text supervision tend to make suboptimal mask predictions when there are text queries referring to non-existing concepts in the image. To alleviate these issues, we introduce a novel recurrent framework that progressively filters out irrelevant texts and enhances mask quality without training efforts. The recurrent unit is a two-stage segmenter built upon a VLM with frozen weights. Thus, our model retains the VLM's broad vocabulary space and strengthens its segmentation capability. Experimental results show that our method outperforms not only the training-free counterparts, but also those fine-tuned with millions of additional data samples, and sets new state-of-the-art records for both zero-shot semantic and referring image segmentation tasks. Specifically, we improve the current record by 28.8, 16.0, and 6.9 mIoU on Pascal VOC, COCO Object, and Pascal Context.

翻译：现有的开放词汇图像分割方法需要在掩码标注和/或图像-文本数据集上进行微调。掩码标签需要大量人工，这限制了分割数据集中类别的数量。因此，预训练视觉语言模型（VLM）的开放词汇能力在微调后严重下降。然而，若不进行微调，在弱图像-文本监督下训练的VLM在处理指向图像中不存在概念的文本查询时，往往会产生次优的掩码预测。为解决这些问题，我们提出一种新颖的循环框架，该框架无需训练即可逐步过滤无关文本并提升掩码质量。循环单元是基于权重冻结的VLM构建的两阶段分割器。因此，我们的模型保留了VLM的广阔词汇空间并强化了分割能力。实验结果表明，我们的方法不仅优于无训练的同类方法，还超越了那些用数百万额外数据样本微调的方法，并在零样本语义分割和指代图像分割任务上均刷新了当前最优记录。具体而言，我们在Pascal VOC、COCO Object和Pascal Context数据集上分别将当前记录提升了28.8、16.0和6.9 mIoU。