In this paper, we address panoramic semantic segmentation which is under-explored due to two critical challenges: (1) image distortions and object deformations on panoramas; (2) lack of semantic annotations in the 360-degree imagery. To tackle these problems, first, we propose the upgraded Transformer for Panoramic Semantic Segmentation, i.e., Trans4PASS+, equipped with Deformable Patch Embedding (DPE) and Deformable MLP (DMLPv2) modules for handling object deformations and image distortions whenever (before or after adaptation) and wherever (shallow or deep levels). Second, we enhance the Mutual Prototypical Adaptation (MPA) strategy via pseudo-label rectification for unsupervised domain adaptive panoramic segmentation. Third, aside from Pinhole-to-Panoramic (Pin2Pan) adaptation, we create a new dataset (SynPASS) with 9,080 panoramic images, facilitating Synthetic-to-Real (Syn2Real) adaptation scheme in 360-degree imagery. Extensive experiments are conducted, which cover indoor and outdoor scenarios, and each of them is investigated with Pin2Pan and Syn2Real regimens. Trans4PASS+ achieves state-of-the-art performances on four domain adaptive panoramic semantic segmentation benchmarks. Code is available at https://github.com/jamycheung/Trans4PASS.

翻译：本文针对全景语义分割问题展开研究，该任务因两大关键挑战而尚未被充分探索：(1) 全景图像中的图像畸变与物体形变；(2) 360度图像缺乏语义标注。为解决这些问题，首先我们提出升级版全景语义分割Transformer——Trans4PASS+，其配备可变形分块嵌入（DPE）与可变形MLP（DMLPv2）模块，可在自适应前后及浅层深层等任意阶段处理物体形变与图像畸变。其次，我们通过伪标签修正增强互原型自适应（MPA）策略，实现无监督域自适应全景分割。再次，除针孔到全景（Pin2Pan）自适应方案外，我们创建包含9,080张全景图像的新数据集SynPASS，推动360度图像的合成到真实（Syn2Real）自适应方案。实验覆盖室内外场景，并分别采用Pin2Pan与Syn2Real两种方案进行验证。Trans4PASS+在四个域自适应全景语义分割基准测试中取得最优性能。代码见https://github.com/jamycheung/Trans4PASS。