We propose integrally pre-trained transformer pyramid network (iTPN), towards jointly optimizing the network backbone and the neck, so that transfer gap between representation models and downstream tasks is minimal. iTPN is born with two elaborated designs: 1) The first pre-trained feature pyramid upon vision transformer (ViT). 2) Multi-stage supervision to the feature pyramid using masked feature modeling (MFM). iTPN is updated to Fast-iTPN, reducing computational memory overhead and accelerating inference through two flexible designs. 1) Token migration: dropping redundant tokens of the backbone while replenishing them in the feature pyramid without attention operations. 2) Token gathering: reducing computation cost caused by global attention by introducing few gathering tokens. The base/large-level Fast-iTPN achieve 88.75%/89.5% top-1 accuracy on ImageNet-1K. With 1x training schedule using DINO, the base/large-level Fast-iTPN achieves 58.4%/58.8% box AP on COCO object detection, and a 57.5%/58.7% mIoU on ADE20K semantic segmentation using MaskDINO. Fast-iTPN can accelerate the inference procedure by up to 70%, with negligible performance loss, demonstrating the potential to be a powerful backbone for downstream vision tasks. The code is available at: github.com/sunsmarterjie/iTPN.

翻译：摘要：我们提出整合预训练Transformer金字塔网络（iTPN），旨在联合优化网络主干与颈部，从而使表征模型与下游任务之间的迁移差距最小化。iTPN具备两项精心设计：1）首个基于视觉Transformer（ViT）的预训练特征金字塔；2）利用掩码特征建模（MFM）对特征金字塔进行多阶段监督。将iTPN升级为Fast-iTPN，通过两项灵活设计降低计算内存开销并加速推理：1）令牌迁移：丢弃主干中的冗余令牌，同时在不使用注意力机制的情况下在特征金字塔中补充令牌；2）令牌聚合：通过引入少量聚合令牌降低全局注意力导致的计算成本。基础/大型Fast-iTPN在ImageNet-1K上分别达到88.75%/89.5%的top-1准确率。采用DINO的一倍训练策略，基础/大型Fast-iTPN在COCO目标检测任务中分别达到58.4%/58.8%的box AP，在ADE20K语义分割任务中使用MaskDINO达到57.5%/58.7%的mIoU。Fast-iTPN可将推理过程加速高达70%，且性能损失可忽略不计，展现出作为下游视觉任务强大主干的潜力。代码已开源在：github.com/sunsmarterjie/iTPN。