Perception is a key building block of autonomously acting vision systems such as autonomous vehicles. It is crucial that these systems are able to understand their surroundings in order to operate safely and robustly. Additionally, autonomous systems deployed in unconstrained real-world scenarios must be able of dealing with novel situations and object that have never been seen before. In this article, we tackle the problem of open-world panoptic segmentation, i.e., the task of discovering new semantic categories and new object instances at test time, while enforcing consistency among the categories that we incrementally discover. We propose Con2MAV, an approach for open-world panoptic segmentation that extends our previous work, ContMAV, which was developed for open-world semantic segmentation. Through extensive experiments across multiple datasets, we show that our model achieves state-of-the-art results on open-world segmentation tasks, while still performing competitively on the known categories. We will open-source our implementation upon acceptance. Additionally, we propose PANIC (Panoptic ANomalies In Context), a benchmark for evaluating open-world panoptic segmentation in autonomous driving scenarios. This dataset, recorded with a multi-modal sensor suite mounted on a car, provides high-quality, pixel-wise annotations of anomalous objects at both semantic and instance level. Our dataset contains 800 images, with more than 50 unknown classes, i.e., classes that do not appear in the training set, and 4000 object instances, making it an extremely challenging dataset for open-world segmentation tasks in the autonomous driving scenario. We provide competitions for multiple open-world tasks on a hidden test set. Our dataset and competitions are available at https://www.ipb.uni-bonn.de/data/panic.

翻译：感知是自动驾驶汽车等自主视觉系统的关键构建模块。为确保系统安全稳健运行，其必须能够理解周围环境。此外，部署在无约束现实场景中的自主系统还需具备处理从未见过的新颖情境和物体的能力。本文致力于解决开放世界全景分割问题，即在测试时发现新的语义类别和新的物体实例，同时确保增量发现类别间的一致性。我们提出Con2MAV方法，该方法扩展了我们先前为开放世界语义分割开发的ContMAV工作，用于开放世界全景分割任务。通过在多个数据集上的广泛实验，我们证明该模型在开放世界分割任务中取得了最先进的结果，同时在已知类别上仍保持竞争力。我们将在论文录用后开源代码。此外，我们提出PANIC（Panoptic ANomalies In Context）基准数据集，用于评估自动驾驶场景中的开放世界全景分割性能。该数据集通过车载多模态传感器套件采集，在语义和实例级别提供异常物体的高质量像素级标注。数据集包含800张图像，涵盖超过50个未见于训练集的未知类别及4000个物体实例，构成自动驾驶场景下极具挑战性的开放世界分割基准。我们针对隐藏测试集设计了多项开放世界任务的竞赛。数据集与竞赛可通过https://www.ipb.uni-bonn.de/data/panic获取。