Unsupervised semantic segmentation aims to categorize each pixel in an image into a corresponding class without the use of annotated data. It is a widely researched area as obtaining labeled datasets is expensive. While previous works in the field have demonstrated a gradual improvement in model accuracy, most required neural network training. This made segmentation equally expensive, especially when dealing with large-scale datasets. We thus propose a lightweight clustering framework for unsupervised semantic segmentation. We discovered that attention features of the self-supervised Vision Transformer exhibit strong foreground-background differentiability. Therefore, clustering can be employed to effectively separate foreground and background image patches. In our framework, we first perform multilevel clustering across the Dataset-level, Category-level, and Image-level, and maintain consistency throughout. Then, the binary patch-level pseudo-masks extracted are upsampled, refined and finally labeled. Furthermore, we provide a comprehensive analysis of the self-supervised Vision Transformer features and a detailed comparison between DINO and DINOv2 to justify our claims. Our framework demonstrates great promise in unsupervised semantic segmentation and achieves state-of-the-art results on PASCAL VOC and MS COCO datasets.

翻译：无监督语义分割旨在无需标注数据的情况下，将图像中的每个像素归类至对应类别。由于获取标注数据集成本高昂，该领域已成为广泛研究的热点。尽管此前研究已逐步提升了模型精度，但多数方法仍需训练神经网络，这使得分割过程同等昂贵，尤其在处理大规模数据集时。为此，我们提出了一种面向无监督语义分割的轻量级聚类框架。研究发现，自监督视觉Transformer的注意力特征具有显著的背景-前景可区分性，因此可通过聚类有效分离前景与背景图像块。该框架首先在数据集层级、类别层级与图像层级实施多层级聚类并保持一致性，随后对提取的二进制块级伪掩码进行上采样、精炼与标注。此外，我们系统分析了自监督视觉Transformer的特征属性，并详细对比了DINO与DINOv2模型以支撑研究结论。本框架在无监督语义分割中展现出显著潜力，并在PASCAL VOC和MS COCO数据集上取得了当前最优性能。