Recently, semantic segmentation models trained with image-level text supervision have shown promising results in challenging open-world scenarios. However, these models still face difficulties in learning fine-grained semantic alignment at the pixel level and predicting accurate object masks. To address this issue, we propose MixReorg, a novel and straightforward pre-training paradigm for semantic segmentation that enhances a model's ability to reorganize patches mixed across images, exploring both local visual relevance and global semantic coherence. Our approach involves generating fine-grained patch-text pairs data by mixing image patches while preserving the correspondence between patches and text. The model is then trained to minimize the segmentation loss of the mixed images and the two contrastive losses of the original and restored features. With MixReorg as a mask learner, conventional text-supervised semantic segmentation models can achieve highly generalizable pixel-semantic alignment ability, which is crucial for open-world segmentation. After training with large-scale image-text data, MixReorg models can be applied directly to segment visual objects of arbitrary categories, without the need for further fine-tuning. Our proposed framework demonstrates strong performance on popular zero-shot semantic segmentation benchmarks, outperforming GroupViT by significant margins of 5.0%, 6.2%, 2.5%, and 3.4% mIoU on PASCAL VOC2012, PASCAL Context, MS COCO, and ADE20K, respectively.

翻译：近期，基于图像级文本监督训练的语义分割模型在具有挑战性的开放世界场景中展现了令人瞩目的成果。然而，这类模型在像素级精细语义对齐学习及精确目标掩码预测方面仍面临困难。为解决此问题，我们提出MixReorg——一种新颖且简洁的语义分割预训练范式。该范式通过增强模型对跨图像混合补丁的重组能力，同时探索局部视觉关联与全局语义一致性。具体而言，我们通过混合图像补丁并保留补丁-文本对应关系，生成细粒度的补丁-文本对数据，随后训练模型最小化混合图像的分割损失、原始特征与重构特征的双重对比损失。以MixReorg作为掩码学习器，传统文本监督语义分割模型可具备高度泛化的像素-语义对齐能力，这对开放世界分割至关重要。经大规模图像-文本数据训练后，MixReorg模型可直接应用于任意类别视觉目标的分割，无需额外微调。所提框架在主流零样本语义分割基准上展现出强劲性能，在PASCAL VOC2012、PASCAL Context、MS COCO及ADE20K数据集上分别超越GroupViT模型达5.0%、6.2%、2.5%和3.4%的mIoU指标。