Semantic segmentation benchmarks in the realm of autonomous driving are dominated by large pre-trained transformers, yet their widespread adoption is impeded by substantial computational costs and prolonged training durations. To lift this constraint, we look at efficient semantic segmentation from a perspective of comprehensive knowledge distillation and consider to bridge the gap between multi-source knowledge extractions and transformer-specific patch embeddings. We put forward the Transformer-based Knowledge Distillation (TransKD) framework which learns compact student transformers by distilling both feature maps and patch embeddings of large teacher transformers, bypassing the long pre-training process and reducing the FLOPs by >85.0%. Specifically, we propose two fundamental and two optimization modules: (1) Cross Selective Fusion (CSF) enables knowledge transfer between cross-stage features via channel attention and feature map distillation within hierarchical transformers; (2) Patch Embedding Alignment (PEA) performs dimensional transformation within the patchifying process to facilitate the patch embedding distillation; (3) Global-Local Context Mixer (GL-Mixer) extracts both global and local information of a representative embedding; (4) Embedding Assistant (EA) acts as an embedding method to seamlessly bridge teacher and student models with the teacher's number of channels. Experiments on Cityscapes, ACDC, NYUv2, and Pascal VOC2012 datasets show that TransKD outperforms state-of-the-art distillation frameworks and rivals the time-consuming pre-training method. The source code is publicly available at https://github.com/RuipingL/TransKD.

翻译：摘要：在自动驾驶领域的语义分割基准测试中，大型预训练Transformer模型占据主导地位，但其广泛应用受限于高昂的计算成本和漫长的训练时长。为突破此瓶颈，本文从全面知识蒸馏的视角审视高效语义分割问题，并致力于弥合多源知识提取与Transformer特有的补丁嵌入机制之间的鸿沟。我们提出基于Transformer的知识蒸馏框架（TransKD），通过蒸馏大型教师Transformer的特征图与补丁嵌入，学习紧凑的学生Transformer模型，从而省去冗长的预训练过程，并将计算量（FLOPs）减少超过85.0%。具体而言，我们提出两个基础模块与两个优化模块：（1）跨选择性融合模块（CSF）通过通道注意力机制实现层级式Transformer中跨阶段特征的知识迁移与特征图蒸馏；（2）补丁嵌入对齐模块（PEA）在补丁分割过程中执行维度变换以促进补丁嵌入蒸馏；（3）全局-局部上下文混合器（GL-Mixer）提取代表性嵌入中的全局与局部信息；（4）嵌入助手模块（EA）作为一种嵌入方法，无缝衔接教师与学生的模型通道数差异。在Cityscapes、ACDC、NYUv2及Pascal VOC2012数据集上的实验表明，TransKD优于现有最优蒸馏框架，并能匹敌耗时的预训练方法。源代码已公开于https://github.com/RuipingL/TransKD。