Commonly used backbones for semantic segmentation, such as ResNet and Swin-Transformer, have multiple stages for feature encoding. Simply using high-resolution low-level feature maps from the early stages of the backbone to directly refine the low-resolution high-level feature map is a common practice of low-resolution feature map upsampling. However, the representation power of the low-level features is generally worse than high-level features, thus introducing ``noise" to the upsampling refinement. To address this issue, we proposed High-level Feature Guided Decoder (HFGD), which uses isolated high-level features to guide low-level features and upsampling process. Specifically, the guidance is realized through carefully designed stop gradient operations and class kernels. Now the class kernels co-evolve only with the high-level features and are reused in the upsampling head to guide the training process of the upsampling head. HFGD is very efficient and effective that can also upsample the feature maps to a previously unseen output stride (OS) of 2 and still obtain accuracy gain. HFGD demonstrates state-of-the-art performance on several benchmark datasets (e.g. Pascal Context, COCOStuff164k and Cityscapes) with small FLOPs. The full code will be available at https://github.com/edwardyehuang/HFGD.git.

翻译：语义分割中常用的骨干网络（如ResNet和Swin-Transformer）具有多级特征编码结构。现有方法通常直接使用骨干网络早期阶段生成的高分辨率低层特征图来优化低分辨率高层特征图，这是低分辨率特征图上采样的常见做法。然而，低层特征的表示能力通常弱于高层特征，因此会在上采样优化过程中引入“噪声”。为解决该问题，我们提出了高层特征引导解码器（HFGD），该方法利用独立的高层特征来引导低层特征及上采样过程。具体而言，通过精心设计的停止梯度操作和类别核实现引导机制。类别核仅与高层特征协同演化，并在上采样头部中复用，从而指导上采样头部的训练过程。HFGD兼具高效性与有效性，能够将特征图上采样至此前未达的输出步长（OS）2，并仍能获得精度提升。实验表明，HFGD在多个基准数据集（如Pascal Context、COCOStuff164k和Cityscapes）上以较小的FLOPs实现了最先进的性能。完整代码将在https://github.com/edwardyehuang/HFGD.git发布。