A surge of interest has emerged in utilizing Transformers in diverse vision tasks owing to its formidable performance. However, existing approaches primarily focus on optimizing internal model architecture designs that often entail significant trial and error with high burdens. In this work, we propose a new paradigm dubbed Decision Stream Calibration that boosts the performance of general Vision Transformers. To achieve this, we shed light on the information propagation mechanism in the learning procedure by exploring the correlation between different tokens and the relevance coefficient of multiple dimensions. Upon further analysis, it was discovered that 1) the final decision is associated with tokens of foreground targets, while token features of foreground target will be transmitted into the next layer as much as possible, and the useless token features of background area will be eliminated gradually in the forward propagation. 2) Each category is solely associated with specific sparse dimensions in the tokens. Based on the discoveries mentioned above, we designed a two-stage calibration scheme, namely ViT-Calibrator, including token propagation calibration stage and dimension propagation calibration stage. Extensive experiments on commonly used datasets show that the proposed approach can achieve promising results. The source codes are given in the supplements.

翻译：由于Transformer在各类视觉任务中展现出卓越性能，近期涌现出大量相关研究。然而现有方法主要聚焦于优化内部模型架构设计，往往需要耗费大量试错成本。本文提出一种名为"决策流校准"的新范式，可提升通用视觉Transformer的性能。为此，我们通过探究不同令牌间的关联性以及多个维度的相关系数，揭示了学习过程中信息传播机制。进一步分析发现：1）最终决策与前景目标的令牌相关，且前景目标令牌特征会尽可能传递至下一层，而背景区域的无用令牌特征在前向传播过程中会被逐步消除；2）每个类别仅与令牌中特定的稀疏维度相关。基于上述发现，我们设计了两阶段校准方案ViT-Calibrator，包含令牌传播校准阶段和维度传播校准阶段。在常用数据集上的大量实验表明，所提方法能取得优异效果。源代码见附录。