Open-vocabulary semantic segmentation strives to distinguish pixels into different semantic groups from an open set of categories. Most existing methods explore utilizing pre-trained vision-language models, in which the key is to adopt the image-level model for pixel-level segmentation task. In this paper, we propose a simple encoder-decoder, named SED, for open-vocabulary semantic segmentation, which comprises a hierarchical encoder-based cost map generation and a gradual fusion decoder with category early rejection. The hierarchical encoder-based cost map generation employs hierarchical backbone, instead of plain transformer, to predict pixel-level image-text cost map. Compared to plain transformer, hierarchical backbone better captures local spatial information and has linear computational complexity with respect to input size. Our gradual fusion decoder employs a top-down structure to combine cost map and the feature maps of different backbone levels for segmentation. To accelerate inference speed, we introduce a category early rejection scheme in the decoder that rejects many no-existing categories at the early layer of decoder, resulting in at most 4.7 times acceleration without accuracy degradation. Experiments are performed on multiple open-vocabulary semantic segmentation datasets, which demonstrates the efficacy of our SED method. When using ConvNeXt-B, our SED method achieves mIoU score of 31.6\% on ADE20K with 150 categories at 82 millisecond ($ms$) per image on a single A6000. We will release it at \url{https://github.com/xb534/SED.git}.

翻译：开放词汇语义分割旨在将像素区分成来自开放类别集合的不同语义组。现有方法多采用预训练的视觉-语言模型，关键在于如何将图像级模型用于像素级分割任务。本文提出一种名为SED的简单编码器-解码器，用于开放词汇语义分割，其包含基于层级编码器的代价图生成模块和带有类别早期拒绝的渐进融合解码器。基于层级编码器的代价图生成模块采用层级主干网络（而非普通Transformer）预测像素级图像-文本代价图。与普通Transformer相比，层级主干网络能更有效捕获局部空间信息，且计算复杂度与输入尺寸呈线性关系。渐进融合解码器采用自上而下结构，融合代价图与不同主干层级的特征图进行分割。为加速推理，我们在解码器中引入类别早期拒绝机制，在解码器浅层拒绝大量不存在的类别，可在不降低精度的情况下实现最高4.7倍加速。在多个开放词汇语义分割数据集上的实验证明了SED方法的有效性。当使用ConvNeXt-B时，我们的SED方法在ADE20K数据集（150个类别）上达到31.6%的mIoU分数，单张A6000显卡处理每幅图像仅需82毫秒。代码将开源至\url{https://github.com/xb534/SED.git}。