Vision transformers have recently made a breakthrough in computer vision showing excellent performance in terms of precision for numerous applications. However, their computational cost is very high compared to alternative approaches such as Convolutional Neural Networks. To address this problem, we propose a novel framework for image classification called RAViT based on a multi-branch network that operates on several copies of the same image with different resolutions to reduce the computational cost while preserving the overall accuracy. Furthermore, our framework includes an early exit mechanism that makes our model adaptive and allows to choose the appropriate trade-off between accuracy and computational cost at run-time. For example in a two-branch architecture, the original image is first resized to reduce its resolution, then a prediction is performed on it using a first transformer and the resulting prediction is reused together with the original-size image to perform a final prediction on a second transformer with less computation than a classical Vision transformer architecture. The early-exit process allows the model to make a final prediction at intermediate branches, saving even more computation. We evaluated our approach on CIFAR-10, Tiny ImageNet, and ImageNet. We obtained an equivalent accuracy to the classical Vision transformer model with only around 70% of FLOPs.

翻译：视觉Transformer近期在计算机视觉领域取得突破，在众多应用中展现出卓越的精度性能。然而，与卷积神经网络等替代方法相比，其计算成本极高。为解决此问题，我们提出了一种名为RAViT的新型图像分类框架，该框架基于多分支网络，对同一图像的不同分辨率副本进行处理，在保持整体精度的同时降低计算成本。此外，我们的框架包含早期退出机制，使模型具备自适应性，并允许在运行时根据需求在精度与计算成本之间选择最佳平衡点。例如在双分支架构中，原始图像首先通过调整尺寸降低分辨率，随后使用第一个Transformer进行预测，所得预测结果将与原始尺寸图像共同输入第二个Transformer进行最终预测，其计算量少于经典视觉Transformer架构。早期退出机制允许模型在中间分支作出最终预测，从而进一步节省计算资源。我们在CIFAR-10、Tiny ImageNet和ImageNet数据集上评估了该方法，仅需约70%的浮点运算量即可获得与经典视觉Transformer模型相当的精度。