This paper investigates the capability of plain Vision Transformers (ViTs) for semantic segmentation using the encoder-decoder framework and introduces \textbf{SegViTv2}. In this study, we introduce a novel Attention-to-Mask (\atm) module to design a lightweight decoder effective for plain ViT. The proposed ATM converts the global attention map into semantic masks for high-quality segmentation results. Our decoder outperforms the popular decoder UPerNet using various ViT backbones while consuming only about $5\%$ of the computational cost. For the encoder, we address the concern of the relatively high computational cost in the ViT-based encoders and propose a \emph{Shrunk++} structure that incorporates edge-aware query-based down-sampling (EQD) and query-based upsampling (QU) modules. The Shrunk++ structure reduces the computational cost of the encoder by up to $50\%$ while maintaining competitive performance. Furthermore, we propose to adapt SegViT for continual semantic segmentation, demonstrating nearly zero forgetting of previously learned knowledge. Experiments show that our proposed SegViTv2 surpasses recent segmentation methods on three popular benchmarks including ADE20k, COCO-Stuff-10k and PASCAL-Context datasets. The code is available through the following link: \url{https://github.com/zbwxp/SegVit}.

翻译：本文研究了普通视觉Transformer（ViT）在编码器-解码器框架下进行语义分割的能力，并提出了\textbf{SegViTv2}。研究中，我们引入了一种新颖的注意力到掩码模块，以设计适用于普通ViT的轻量级解码器。所提出的ATM将全局注意力图转换为语义掩码，从而获得高质量的分割结果。我们的解码器在使用多种ViT骨干网络时，性能优于流行的UPerNet解码器，而计算成本仅为其约$5\%$。对于编码器部分，我们解决了基于ViT的编码器中计算成本相对较高的问题，并提出了一种\textit{Shrunk++}结构，该结构融合了边缘感知的基于查询的下采样模块和基于查询的上采样模块。Shrunk++结构将编码器的计算成本降低高达$50\%$，同时保持具有竞争力的性能。此外，我们提出将SegViT适配于连续语义分割任务，展示出对先前学习知识的近乎零遗忘。实验表明，我们提出的SegViTv2在包括ADE20k、COCO-Stuff-10k和PASCAL-Context数据集在内的三个流行基准上超越了最新的分割方法。代码可通过以下链接获取：\url{https://github.com/zbwxp/SegVit}。