Current state-of-the-art methods for panoptic segmentation require an immense amount of annotated training data that is both arduous and expensive to obtain posing a significant challenge for their widespread adoption. Concurrently, recent breakthroughs in visual representation learning have sparked a paradigm shift leading to the advent of large foundation models that can be trained with completely unlabeled images. In this work, we propose to leverage such task-agnostic image features to enable few-shot panoptic segmentation by presenting Segmenting Panoptic Information with Nearly 0 labels (SPINO). In detail, our method combines a DINOv2 backbone with lightweight network heads for semantic segmentation and boundary estimation. We show that our approach, albeit being trained with only ten annotated images, predicts high-quality pseudo-labels that can be used with any existing panoptic segmentation method. Notably, we demonstrate that SPINO achieves competitive results compared to fully supervised baselines while using less than 0.3% of the ground truth labels, paving the way for learning complex visual recognition tasks leveraging foundation models. To illustrate its general applicability, we further deploy SPINO on real-world robotic vision systems for both outdoor and indoor environments. To foster future research, we make the code and trained models publicly available at http://spino.cs.uni-freiburg.de.

翻译：当前最先进的全景分割方法需要大量标注训练数据，这些数据的获取既艰巨又昂贵，对其广泛应用构成了重大挑战。与此同时，视觉表征学习的最新突破引发了范式变革，催生了可在完全无标注图像上训练的大型基础模型。本文提出利用此类任务无关图像特征实现少样本全景分割，具体方法为SPINO（近乎零标注的全景信息分割）。我们的方法将DINOv2骨干网络与轻量级语义分割和边界估计网络头相结合。研究表明，尽管仅使用十张标注图像训练，该方法仍能生成高质量伪标签，这些标签可用于任何现有全景分割方法。值得注意的是，我们证明SPINO在使用不到0.3%真实标注数据的情况下，即可达到与全监督基线相媲美的竞争性结果，为基础模型驱动复杂视觉识别任务学习开辟了新路径。为展示其通用适用性，我们进一步将SPINO部署于户外及室内真实机器人视觉系统。为推动后续研究，代码与预训练模型已在http://spino.cs.uni-freiburg.de公开。