Due to spatial redundancy in remote sensing images, sparse tokens containing rich information are usually involved in self-attention (SA) to reduce the overall token numbers within the calculation, avoiding the high computational cost issue in Vision Transformers. However, such methods usually obtain sparse tokens by hand-crafted or parallel-unfriendly designs, posing a challenge to reach a better balance between efficiency and performance. Different from them, this paper proposes to use learnable meta tokens to formulate sparse tokens, which effectively learn key information meanwhile improving the inference speed. Technically, the meta tokens are first initialized from image tokens via cross-attention. Then, we propose Dual Cross-Attention (DCA) to promote information exchange between image tokens and meta tokens, where they serve as query and key (value) tokens alternatively in a dual-branch structure, significantly reducing the computational complexity compared to self-attention. By employing DCA in the early stages with dense visual tokens, we obtain the hierarchical architecture LeMeViT with various sizes. Experimental results in classification and dense prediction tasks show that LeMeViT has a significant $1.7 \times$ speedup, fewer parameters, and competitive performance compared to the baseline models, and achieves a better trade-off between efficiency and performance.

翻译：针对遥感图像中的空间冗余问题，现有方法通常通过自注意力机制引入包含丰富信息的稀疏令牌以减少计算中的令牌总数，从而规避视觉Transformer的高计算开销。然而，这类方法通常依赖人工设计或非并行友好的方式获取稀疏令牌，难以实现效率与性能的平衡。本文提出采用可学习元令牌构建稀疏令牌，在有效学习关键信息的同时提升推理速度。技术层面，元令牌首先通过交叉注意力从图像令牌初始化，进而提出双分支交叉注意力机制促进图像令牌与元令牌间的信息交互——两者在双分支结构中交替作为查询与键（值）令牌，较之自注意力显著降低计算复杂度。通过在早期阶段结合密集视觉令牌使用双分支交叉注意力，我们构建了多尺度层级架构LeMeViT。在分类与密集预测任务上的实验结果表明，相比基线模型，LeMeViT在参数量更少的前提下实现了1.7倍的推理加速，并展现出具有竞争力的性能表现，达成了效率与性能的更优权衡。