Panoptic segmentation methods assign a known class to each pixel given in input. Even for state-of-the-art approaches, this inevitably enforces decisions that systematically lead to wrong predictions for objects outside the training categories. However, robustness against out-of-distribution samples and corner cases is crucial in safety-critical settings to avoid dangerous consequences. Since real-world datasets cannot contain enough data points to adequately sample the long tail of the underlying distribution, models must be able to deal with unseen and unknown scenarios as well. Previous methods targeted this by re-identifying already-seen unlabeled objects. In this work, we propose the necessary step to extend segmentation with a new setting which we term holistic segmentation. Holistic segmentation aims to identify and separate objects of unseen, unknown categories into instances without any prior knowledge about them while performing panoptic segmentation of known classes. We tackle this new problem with U3HS, which finds unknowns as highly uncertain regions and clusters their corresponding instance-aware embeddings into individual objects. By doing so, for the first time in panoptic segmentation with unknown objects, our U3HS is trained without unknown categories, reducing assumptions and leaving the settings as unconstrained as in real-life scenarios. Extensive experiments on public data from MS COCO, Cityscapes, and Lost&Found demonstrate the effectiveness of U3HS for this new, challenging, and assumptions-free setting called holistic segmentation. Project page: https://holisticseg.github.io.

翻译：全景分割方法为输入中的每个像素分配一个已知类别。即便是最先进的方法，也难免对训练类别之外的物体做出系统性误判。然而，在安全关键场景中，对分布外样本和边缘情况的鲁棒性至关重要，以避免危险后果。由于现实数据集无法包含足够的数据点以充分采样底层分布的长尾部分，模型必须能够处理未见和未知场景。以往方法通过重新识别已见过的未标注目标来解决这一问题。本工作中，我们提出必要步骤，将分割扩展至一种新设置，即全息分割。全息分割旨在识别并分离未见、未知类别中的物体为实例，无需任何关于它们的先验知识，同时对已知类别执行全景分割。我们使用U3HS方法解决这一新问题，该方法将未知区域识别为高度不确定区域，并将其对应的实例感知嵌入聚类为单个物体。通过此方式，首次在包含未知物体的全景分割中，我们的U3HS无需未知类别即可训练，减少了假设条件，使设置与现实场景一样无约束。在MS COCO、Cityscapes和Lost&Found公开数据集上的大量实验证明了U3HS对于这一全新、富有挑战性且无假设的“全息分割”设置的有效性。项目页面：https://holisticseg.github.io。