This work presents Adaptive Local-then-Global Merging (ALGM), a token reduction method for semantic segmentation networks that use plain Vision Transformers. ALGM merges tokens in two stages: (1) In the first network layer, it merges similar tokens within a small local window and (2) halfway through the network, it merges similar tokens across the entire image. This is motivated by an analysis in which we found that, in those situations, tokens with a high cosine similarity can likely be merged without a drop in segmentation quality. With extensive experiments across multiple datasets and network configurations, we show that ALGM not only significantly improves the throughput by up to 100%, but can also enhance the mean IoU by up to +1.1, thereby achieving a better trade-off between segmentation quality and efficiency than existing methods. Moreover, our approach is adaptive during inference, meaning that the same model can be used for optimal efficiency or accuracy, depending on the application. Code is available at https://tue-mps.github.io/ALGM.

翻译：本文提出了自适应局部-全局合并（ALGM）方法，这是一种面向采用朴素视觉Transformer的语义分割网络的令牌约简方法。ALGM分两个阶段合并令牌：（1）在网络第一层中，该方法在局部小窗口内合并相似令牌；（2）在网络中间层，该方法在整个图像范围内合并相似令牌。该设计的动机源于我们的分析发现：在上述场景中，具有高余弦相似度的令牌进行合并后，分割质量很可能不会下降。通过在多个数据集和网络配置上的大量实验，我们证明ALGM不仅将处理吞吐量最高提升100%，还能将平均交并比（mIoU）最高提升+1.1，从而在分割质量与效率之间实现了比现有方法更优的权衡。此外，我们的方法在推理过程中具有自适应性，这意味着同一模型可根据应用需求灵活调整为最优效率模式或最优精度模式。代码发布于 https://tue-mps.github.io/ALGM。