Recently, the tokens of images share the same static data flow in many dense networks. However, challenges arise from the variance among the objects in images, such as large variations in the spatial scale and difficulties of recognition for visual entities. In this paper, we propose a data-dependent token routing strategy to elaborate the routing paths of image tokens for Dynamic Vision Transformer, dubbed DiT. The proposed framework generates a data-dependent path per token, adapting to the object scales and visual discrimination of tokens. In feed-forward, the differentiable routing gates are designed to select the scaling paths and feature transformation paths for image tokens, leading to multi-path feature propagation. In this way, the impact of object scales and visual discrimination of image representation can be carefully tuned. Moreover, the computational cost can be further reduced by giving budget constraints to the routing gate and early-stopping of feature extraction. In experiments, our DiT achieves superior performance and favorable complexity/accuracy trade-offs than many SoTA methods on ImageNet classification, object detection, instance segmentation, and semantic segmentation. Particularly, the DiT-B5 obtains 84.8\% top-1 Acc on ImageNet with 10.3 GFLOPs, which is 1.0\% higher than that of the SoTA method with similar computational complexity. These extensive results demonstrate that DiT can serve as versatile backbones for various vision tasks.

翻译：近期，许多密集网络中图像令牌共享相同的静态数据流。然而，图像中物体间的差异性（如空间尺度的大幅变化及视觉实体识别难度）带来了挑战。本文提出一种数据依赖的令牌路由策略，为动态视觉Transformer（Dynamic Vision Transformer）设计精细化的令牌路径，称为DiT。该框架为每个令牌生成数据依赖路径，自适应匹配物体尺度与令牌视觉判别性。在前馈过程中，可微分路由门被设计用于选择图像令牌的缩放路径与特征变换路径，实现多路径特征传播。通过此机制，物体尺度与视觉判别性对图像表征的影响可被精细调控。此外，通过为路由门设置预算约束并提前终止特征提取，可进一步降低计算成本。实验表明，在ImageNet分类、目标检测、实例分割及语义分割任务中，DiT相比诸多现有最优方法（SoTA）实现了优越性能与理想的复杂度/精度权衡。尤其值得注意的是，DiT-B5在ImageNet上以10.3 GFLOPs计算量达到84.8%的Top-1准确率，较同等计算复杂度的SoTA方法提升1.0%。这些广泛结果证明DiT可作为多种视觉任务的多功能骨干网络。