A surge of interest has emerged in utilizing Transformers in diverse vision tasks owing to its formidable performance. However, existing approaches primarily focus on optimizing internal model architecture designs that often entail significant trial and error with high burdens. In this work, we propose a new paradigm dubbed Decision Stream Calibration that boosts the performance of general Vision Transformers. To achieve this, we shed light on the information propagation mechanism in the learning procedure by exploring the correlation between different tokens and the relevance coefficient of multiple dimensions. Upon further analysis, it was discovered that 1) the final decision is associated with tokens of foreground targets, while token features of foreground target will be transmitted into the next layer as much as possible, and the useless token features of background area will be eliminated gradually in the forward propagation. 2) Each category is solely associated with specific sparse dimensions in the tokens. Based on the discoveries mentioned above, we designed a two-stage calibration scheme, namely ViT-Calibrator, including token propagation calibration stage and dimension propagation calibration stage. Extensive experiments on commonly used datasets show that the proposed approach can achieve promising results. The source codes are given in the supplements.

翻译：由于Transformer在各类视觉任务中展现出强大性能，学界对其应用产生了浓厚兴趣。然而现有方法主要关注内部模型架构设计的优化，往往需要高成本的反复试验。本文提出名为"决策流校准"的新范式，用于提升通用视觉Transformer的性能。为此，我们通过探索不同标记间的相关性与多维度关联系数，揭示了学习过程中的信息传播机制。进一步分析发现：1）最终决策与前景目标标记相关联，前向传播中前景目标标记特征将尽可能传递至下一层，而背景区域的无用标记特征会逐步消除；2）每个类别仅与标记中特定的稀疏维度相关联。基于上述发现，我们设计了包含标记传播校准阶段和维度传播校准阶段的两阶段校准方案ViT-Calibrator。在常用数据集上的大量实验表明，该方法能够取得理想结果。源代码详见附录。