Panoptic segmentation methods assign a known class to each pixel given in input. Even for state-of-the-art approaches, this inevitably enforces decisions that systematically lead to wrong predictions for objects outside the training categories. However, robustness against out-of-distribution samples and corner cases is crucial in safety-critical settings to avoid dangerous consequences. Since real-world datasets cannot contain enough data points to adequately sample the long tail of the underlying distribution, models must be able to deal with unseen and unknown scenarios as well. Previous methods targeted this by re-identifying already-seen unlabeled objects. In this work, we propose the necessary step to extend segmentation with a new setting which we term holistic segmentation. Holistic segmentation aims to identify and separate objects of unseen unknown categories into instances, without any prior knowledge about them, while performing panoptic segmentation of known classes. We tackle this new problem with U3HS, which finds unknowns as highly uncertain regions and clusters their corresponding instance-aware embeddings into individual objects. By doing so, for the first time in panoptic segmentation with unknown objects, our U3HS is trained without unknown categories, reducing assumptions and leaving the settings as unconstrained as in real-life scenarios. Extensive experiments on public data from MS COCO, Cityscapes, and Lost&Found demonstrate the effectiveness of U3HS for this new, challenging, and assumptions-free setting called holistic segmentation.

翻译：全景分割方法为输入中的每个像素分配已知类别。即便是最先进的方法，也必然导致对训练类别之外的物体做出错误预测。然而，在安全关键场景中，对分布外样本和极端情况的鲁棒性至关重要，以避免危险后果。由于真实世界数据集无法包含足够的数据点来充分采样底层分布的长尾部分，模型必须能够处理未见过的未知场景。先前的方法通过重新识别已见过的未标记对象来应对这一问题。在本工作中，我们提出必要的一步，将分割扩展到一个新设定，称为整体分割。整体分割旨在识别和分离未知类别的物体为实例，无需任何关于它们的先验知识，同时对已知类别执行全景分割。我们通过U3HS解决这一新问题，该方法将未知物视为高不确定性区域，并将其对应的实例感知嵌入聚类为独立物体。通过这样做，U3HS首次在没有未知类别的情况下进行未知物体的全景分割训练，减少了假设条件，使设定与现实场景一样无约束。在来自MS COCO、Cityscapes和Lost&Found的公共数据上进行的大量实验证明了U3HS在这种无假设的新挑战性设定——整体分割中的有效性。