In this paper, we address panoramic semantic segmentation which is under-explored due to two critical challenges: (1) image distortions and object deformations on panoramas; (2) lack of semantic annotations in the 360-degree imagery. To tackle these problems, first, we propose the upgraded Transformer for Panoramic Semantic Segmentation, i.e., Trans4PASS+, equipped with Deformable Patch Embedding (DPE) and Deformable MLP (DMLPv2) modules for handling object deformations and image distortions whenever (before or after adaptation) and wherever (shallow or deep levels). Second, we enhance the Mutual Prototypical Adaptation (MPA) strategy via pseudo-label rectification for unsupervised domain adaptive panoramic segmentation. Third, aside from Pinhole-to-Panoramic (Pin2Pan) adaptation, we create a new dataset (SynPASS) with 9,080 panoramic images, facilitating Synthetic-to-Real (Syn2Real) adaptation scheme in 360-degree imagery. Extensive experiments are conducted, which cover indoor and outdoor scenarios, and each of them is investigated with Pin2Pan and Syn2Real regimens. Trans4PASS+ achieves state-of-the-art performances on four domain adaptive panoramic semantic segmentation benchmarks. Code is available at https://github.com/jamycheung/Trans4PASS.

翻译：本文针对全景语义分割展开研究，该任务因两大关键挑战尚未充分探索：（1）全景图像中的畸变与物体形变；（2）360度图像缺乏语义标注。为解决这些问题，我们首先提出升级版全景语义分割Transformer——Trans4PASS+，配备可变形块嵌入（DPE）与可变形MLP（DMLPv2）模块，可在适配前后及深浅层等任意位置处理物体形变与图像畸变。其次，通过伪标签修正增强互原型自适应（MPA）策略，实现无监督域自适应全景分割。第三，除针孔到全景（Pin2Pan）适配外，我们构建包含9,080张全景图像的新数据集SynPASS，推动360度图像中的合成到真实（Syn2Real）适配方案。实验覆盖室内外场景，并分别采用Pin2Pan与Syn2Real方案进行研究。Trans4PASS+在四个域自适应全景语义分割基准上达到最优性能。代码开源地址：https://github.com/jamycheung/Trans4PASS。