We present PVO, a novel panoptic visual odometry framework to achieve more comprehensive modeling of the scene motion, geometry, and panoptic segmentation information. Our PVO models visual odometry (VO) and video panoptic segmentation (VPS) in a unified view, which makes the two tasks mutually beneficial. Specifically, we introduce a panoptic update module into the VO Module with the guidance of image panoptic segmentation. This Panoptic-Enhanced VO Module can alleviate the impact of dynamic objects in the camera pose estimation with a panoptic-aware dynamic mask. On the other hand, the VO-Enhanced VPS Module also improves the segmentation accuracy by fusing the panoptic segmentation result of the current frame on the fly to the adjacent frames, using geometric information such as camera pose, depth, and optical flow obtained from the VO Module. These two modules contribute to each other through recurrent iterative optimization. Extensive experiments demonstrate that PVO outperforms state-of-the-art methods in both visual odometry and video panoptic segmentation tasks.

翻译：我们提出PVO，一种新颖的全景视觉里程计框架，旨在更全面地建模场景运动、几何以及全景分割信息。我们的PVO将视觉里程计（VO）与视频全景分割（VPS）统一建模，使两个任务相互增益。具体而言，我们在图像全景分割的引导下，为VO模块引入了一个全景更新模块。这种全景增强型VO模块能够利用全景感知动态掩码减轻动态物体对相机位姿估计的影响。另一方面，VO增强型VPS模块通过将当前帧的全景分割结果实时融合到相邻帧，并利用从VO模块获取的相机位姿、深度和光流等几何信息，提升了分割精度。这两个模块通过循环迭代优化相互促进。大量实验表明，PVO在视觉里程计和视频全景分割任务中均优于现有最先进方法。