The recently proposed MaskFormer gives a refreshed perspective on the task of semantic segmentation: it shifts from the popular pixel-level classification paradigm to a mask-level classification method. In essence, it generates paired probabilities and masks corresponding to category segments and combines them during inference for the segmentation maps. In our study, we find that per-mask classification decoder on top of a single-scale feature is not effective enough to extract reliable probability or mask. To mine for rich semantic information across the feature pyramid, we propose a transformer-based Pyramid Fusion Transformer (PFT) for per-mask approach semantic segmentation with multi-scale features. The proposed transformer decoder performs cross-attention between the learnable queries and each spatial feature from the feature pyramid in parallel and uses cross-scale inter-query attention to exchange complimentary information. We achieve competitive performance on three widely used semantic segmentation datasets. In particular, on ADE20K validation set, our result with Swin-B backbone surpasses that of MaskFormer's with a much larger Swin-L backbone in both single-scale and multi-scale inference, achieving 54.1 mIoU and 55.7 mIoU respectively. Using a Swin-L backbone, we achieve single-scale 56.1 mIoU and multi-scale 57.4 mIoU, obtaining state-of-the-art performance on the dataset. Extensive experiments on three widely used semantic segmentation datasets verify the effectiveness of our proposed method.

翻译：近期提出的MaskFormer为语义分割任务提供了全新视角：它将主流的像素级分类范式转向掩码级分类方法。本质上，该方法生成与类别片段对应的配对概率和掩码，并在推理阶段融合得到分割图。本研究发现，基于单尺度特征构建的逐掩码分类解码器难以有效提取可靠的概率或掩码信息。为挖掘跨特征金字塔的丰富语义信息，我们提出基于Transformer的金字塔融合Transformer（Pyramid Fusion Transformer, PFT），实现多尺度特征的逐掩码语义分割方法。所提出的Transformer解码器并行执行可学习查询与特征金字塔中各空间特征的交叉注意力，并通过跨尺度查询间注意力交换互补信息。我们在三个广泛使用的语义分割数据集上取得了竞争性结果。特别地，在ADE20K验证集上，使用Swin-B骨干网络的模型在单尺度与多尺度推理中均超越采用更大规模Swin-L骨干网络的MaskFormer，分别达到54.1 mIoU和55.7 mIoU。采用Swin-L骨干网络时，单尺度与多尺度推理分别实现56.1 mIoU和57.4 mIoU，在该数据集上取得最优性能。在三个广泛使用的语义分割数据集上的大量实验验证了所提方法的有效性。