In this paper, we address panoramic semantic segmentation which is under-explored due to two critical challenges: (1) image distortions and object deformations on panoramas; (2) lack of semantic annotations in the 360{\deg} imagery. To tackle these problems, first, we propose the upgraded Transformer for Panoramic Semantic Segmentation, i.e., Trans4PASS+, equipped with Deformable Patch Embedding (DPE) and Deformable MLP (DMLPv2) modules for handling object deformations and image distortions whenever (before or after adaptation) and wherever (shallow or deep levels). Second, we enhance the Mutual Prototypical Adaptation (MPA) strategy via pseudo-label rectification for unsupervised domain adaptive panoramic segmentation. Third, aside from Pinhole-to-Panoramic (Pin2Pan) adaptation, we create a new dataset (SynPASS) with 9,080 panoramic images, facilitating Synthetic-to-Real (Syn2Real) adaptation scheme in 360{\deg} imagery. Extensive experiments are conducted, which cover indoor and outdoor scenarios, and each of them is investigated with Pin2Pan and Syn2Real regimens. Trans4PASS+ achieves state-of-the-art performances on four domain adaptive panoramic semantic segmentation benchmarks. Code is available at https://github.com/jamycheung/Trans4PASS.

翻译：本文针对全景语义分割这一研究不足的领域展开研究，该领域面临两个关键挑战：(1) 全景图像中的图像畸变与物体形变；(2) 360°影像中语义标注数据的缺乏。为解决这些问题，我们首先提出了全景语义分割增强型Transformer——Trans4PASS+，该模型配备可变形块嵌入模块与可变形多层感知机模块，能够在任意时机与任意网络层级处理物体形变与图像畸变。其次，我们通过伪标签校正机制改进了互原型适配策略，以提升无监督领域自适应全景分割的性能。第三，除针孔相机到全景相机的适配范式外，我们构建了包含9,080张全景图像的新数据集，为360°影像中的合成到真实场景适配研究提供支持。我们在室内外场景中开展了大量实验，分别通过针孔相机到全景相机与合成到真实两种适配方案进行验证。Trans4PASS+在四个领域自适应全景语义分割基准测试中均取得了最先进的性能。代码已开源。