Video Panoptic Segmentation (VPS) aims to achieve comprehensive pixel-level scene understanding by segmenting all pixels and associating objects in a video. Current solutions can be categorized into online and near-online approaches. Evolving over the time, each category has its own specialized designs, making it nontrivial to adapt models between different categories. To alleviate the discrepancy, in this work, we propose a unified approach for online and near-online VPS. The meta architecture of the proposed Video-kMaX consists of two components: within clip segmenter (for clip-level segmentation) and cross-clip associater (for association beyond clips). We propose clip-kMaX (clip k-means mask transformer) and HiLA-MB (Hierarchical Location-Aware Memory Buffer) to instantiate the segmenter and associater, respectively. Our general formulation includes the online scenario as a special case by adopting clip length of one. Without bells and whistles, Video-kMaX sets a new state-of-the-art on KITTI-STEP and VIPSeg for video panoptic segmentation, and VSPW for video semantic segmentation. Code will be made publicly available.

翻译：视频全景分割（VPS）旨在通过分割视频中所有像素并关联目标，实现全面的像素级场景理解。现有方法可分为在线与近在线两类。随着时间推移，每类方法均发展出特定设计，使得不同类别间的模型适配具有挑战性。为消除此差异，本文提出一种适用于在线与近在线VPS的统一方法。所提出的Video-kMaX元架构包含两个组件：片段内分割器（用于片段级分割）和跨片段关联器（用于片段间关联）。我们分别提出clip-kMaX（片段级k均值掩码Transformer）和HiLA-MB（分层位置感知记忆缓冲区）作为分割器与关联器的具体实现。通过将片段长度设为1，该通用框架将在线场景作为特例包含其中。无需额外技巧，Video-kMaX在KITTI-STEP和VIPSeg的视频全景分割任务以及VSPW的视频语义分割任务中均达到最新最优性能。代码将开源。