Pure transformers have shown great potential for vision tasks recently. However, their accuracy in small or medium datasets is not satisfactory. Although some existing methods introduce a CNN as a teacher to guide the training process by distillation, the gap between teacher and student networks would lead to sub-optimal performance. In this work, we propose a new One-shot Vision transformer search framework with Online distillation, namely OVO. OVO samples sub-nets for both teacher and student networks for better distillation results. Benefiting from the online distillation, thousands of subnets in the supernet are well-trained without extra finetuning or retraining. In experiments, OVO-Ti achieves 73.32% top-1 accuracy on ImageNet and 75.2% on CIFAR-100, respectively.

翻译：纯Transformer近年来在视觉任务中展现出巨大潜力，但在中小型数据集上的准确率尚不理想。尽管现有方法通过引入CNN教师网络指导学生网络训练的蒸馏机制，但师生网络间的差异会导致次优性能。本文提出一种基于在线蒸馏的新型一次性视觉Transformer搜索框架OVO。该框架通过同时为师生网络采样子网络，实现更优蒸馏效果。得益于在线蒸馏机制，超网络中的数千个子网络无需额外微调或重训练即可充分训练。实验结果表明，OVO-Ti在ImageNet上达到73.32%的Top-1准确率，在CIFAR-100上达到75.2%。