The emergence of vision transformers (ViTs) in image classification has shifted the methodologies for visual representation learning. In particular, ViTs learn visual representation at full receptive field per layer across all the image patches, in comparison to the increasing receptive fields of CNNs across layers and other alternatives (e.g., large kernels and atrous convolution). In this work, for the first time we explore the global context learning potentials of ViTs for dense visual prediction (e.g., semantic segmentation). Our motivation is that through learning global context at full receptive field layer by layer, ViTs may capture stronger long-range dependency information, critical for dense prediction tasks. We first demonstrate that encoding an image as a sequence of patches, a vanilla ViT without local convolution and resolution reduction can yield stronger visual representation for semantic segmentation. For example, our model, termed as SEgmentation TRansformer (SETR), excels on ADE20K (50.28% mIoU, the first position in the test leaderboard on the day of submission) and Pascal Context (55.83% mIoU), and performs competitively on Cityscapes. For tackling general dense visual prediction tasks in a cost-effective manner, we further formulate a family of Hierarchical Local-Global (HLG) Transformers, characterized by local attention within windows and global-attention across windows in a pyramidal architecture. Extensive experiments show that our methods achieve appealing performance on a variety of dense prediction tasks (e.g., object detection and instance segmentation and semantic segmentation) as well as image classification. Our code and models are available at https://github.com/fudan-zvg/SETR.

翻译：视觉Transformer（ViTs）在图像分类领域的出现，彻底改变了视觉表征学习的方法论。具体而言，ViTs在所有图像块的每一层中都能以完整感受野学习视觉表征，而相比之下，CNN的感受野需逐层递增，且其他替代方案（如大卷积核和空洞卷积）也存在局限。本研究首次探索了ViTs在密集视觉预测（如语义分割）中的全局上下文学习潜力。我们的动机在于：通过逐层以完整感受野学习全局上下文，ViTs能够捕获更强的长程依赖信息，这对密集预测任务至关重要。我们首先证明，将图像编码为分块序列后，一个不含局部卷积和分辨率缩减的原始ViT仍能为语义分割生成更强的视觉表征。例如，我们所提出的模型——分割Transformer（SETR），在ADE20K（mIoU达50.28%，提交当日测试排行榜首位）和Pascal Context（mIoU达55.83%）数据集上表现卓越，并在Cityscapes数据集上具备竞争力。为高效处理通用密集视觉预测任务，我们进一步构建了层次化局部-全局（HLG）Transformer家族，其特点在于金字塔结构中窗口内的局部注意力与跨窗口的全局注意力。大量实验表明，我们的方法在多种密集预测任务（如目标检测、实例分割及语义分割）以及图像分类中均取得了优异性能。相关代码与模型已开源至https://github.com/fudan-zvg/SETR。