Deep learning has led to remarkable strides in scene understanding with panoptic segmentation emerging as a key holistic scene interpretation task. However, the performance of panoptic segmentation is severely impacted in the presence of out-of-distribution (OOD) objects i.e. categories of objects that deviate from the training distribution. To overcome this limitation, we propose Panoptic Out-of Distribution Segmentation for joint pixel-level semantic in-distribution and out-of-distribution classification with instance prediction. We extend two established panoptic segmentation benchmarks, Cityscapes and BDD100K, with out-of-distribution instance segmentation annotations, propose suitable evaluation metrics, and present multiple strong baselines. Importantly, we propose the novel PoDS architecture with a shared backbone, an OOD contextual module for learning global and local OOD object cues, and dual symmetrical decoders with task-specific heads that employ our alignment-mismatch strategy for better OOD generalization. Combined with our data augmentation strategy, this approach facilitates progressive learning of out-of-distribution objects while maintaining in-distribution performance. We perform extensive evaluations that demonstrate that our proposed PoDS network effectively addresses the main challenges and substantially outperforms the baselines. We make the dataset, code, and trained models publicly available at http://pods.cs.uni-freiburg.de.

翻译：深度学习在全景分割这一关键的整体场景理解任务中取得了显著进展。然而，当存在分布外对象（即偏离训练分布的物体类别）时，全景分割的性能会受到严重影响。为克服这一局限，我们提出了全场景分布外分割方法，用于联合像素级语义分布内与分布外分类及实例预测。我们扩展了两个已建立的全景分割基准数据集Cityscapes和BDD100K，补充了分布外实例分割标注，提出了适用的评估指标，并构建了多个强基线模型。重要的是，我们提出了新颖的PoDS架构，该架构包含共享骨干网络、用于学习全局与局部分布外对象线索的分布外上下文模块，以及采用我们设计的对齐-失配策略以提升分布外泛化能力的双对称解码器（配备任务专用头）。结合我们的数据增强策略，该方法能够在保持分布内性能的同时促进分布外对象的渐进式学习。我们进行了广泛评估，表明所提出的PoDS网络有效解决了主要挑战，并显著优于基线模型。我们将数据集、代码及训练模型公开于http://pods.cs.uni-freiburg.de。