In view of the fact that semi- and self-supervised learning share a fundamental principle, effectively modeling knowledge from unlabeled data, various semi-supervised semantic segmentation methods have integrated representative self-supervised learning paradigms for further regularization. However, the potential of the state-of-the-art generative self-supervised paradigm, masked image modeling, has been scarcely studied. This paradigm learns the knowledge through establishing connections between the masked and visible parts of masked image, during the pixel reconstruction process. By inheriting and extending this insight, we successfully leverage masked image modeling to boost semi-supervised semantic segmentation. Specifically, we introduce a novel class-wise masked image modeling that independently reconstructs different image regions according to their respective classes. In this way, the mask-induced connections are established within each class, mitigating the semantic confusion that arises from plainly reconstructing images in basic masked image modeling. To strengthen these intra-class connections, we further develop a feature aggregation strategy that minimizes the distances between features corresponding to the masked and visible parts within the same class. Additionally, in semantic space, we explore the application of masked image modeling to enhance regularization. Extensive experiments conducted on well-known benchmarks demonstrate that our approach achieves state-of-the-art performance. The code will be available at https://github.com/haoxt/S4MIM.

翻译：鉴于半监督与自监督学习共享一个核心原则——即有效建模未标记数据中的知识，多种半监督语义分割方法已整合代表性的自监督学习范式以增强正则化效果。然而，当前最先进的生成式自监督范式——掩码图像建模的潜力尚未得到充分探索。该范式通过在像素重建过程中建立掩码图像中被遮蔽部分与可见部分之间的联系来学习知识。通过继承并拓展这一思路，我们成功利用掩码图像建模来提升半监督语义分割的性能。具体而言，我们提出了一种新颖的按类别掩码图像建模方法，该方法根据图像区域所属的不同类别独立重建各区域。通过这种方式，掩码诱导的联系在各类别内部建立，缓解了基础掩码图像建模中直接重建图像可能引发的语义混淆问题。为强化这些类内联系，我们进一步开发了一种特征聚合策略，旨在最小化同一类别内掩码部分与可见部分对应特征之间的距离。此外，在语义空间中，我们探索了应用掩码图像建模以增强正则化的方法。在多个知名基准数据集上的大量实验表明，我们的方法实现了最先进的性能。代码将在 https://github.com/haoxt/S4MIM 公开。