In previous deep-learning-based methods, semantic segmentation has been regarded as a static or dynamic per-pixel classification task, \textit{i.e.,} classify each pixel representation to a specific category. However, these methods only focus on learning better pixel representations or classification kernels while ignoring the structural information of objects, which is critical to human decision-making mechanism. In this paper, we present a new paradigm for semantic segmentation, named structure-aware extraction. Specifically, it generates the segmentation results via the interactions between a set of learned structure tokens and the image feature, which aims to progressively extract the structural information of each category from the feature. Extensive experiments show that our StructToken outperforms the state-of-the-art on three widely-used benchmarks, including ADE20K, Cityscapes, and COCO-Stuff-10K.

翻译：在以往的深度学习方法中，语义分割被视为一种静态或动态的逐像素分类任务，即将每个像素表征分类至特定类别。然而，这些方法仅专注于学习更好的像素表征或分类核，而忽略了物体的结构信息——这一信息对人类决策机制至关重要。本文提出一种新的语义分割范式，名为结构感知提取。具体而言，该方法通过一组学习到的结构令牌与图像特征之间的交互生成分割结果，旨在逐步从特征中提取每个类别的结构信息。大量实验表明，我们的StructToken在包括ADE20K、Cityscapes和COCO-Stuff-10K在内的三个广泛使用的基准测试中均优于当前最先进方法。