Vision transformers have been applied successfully for image recognition tasks. There have been either multi-headed self-attention based (ViT \cite{dosovitskiy2020image}, DeIT, \cite{touvron2021training}) similar to the original work in textual models or more recently based on spectral layers (Fnet\cite{lee2021fnet}, GFNet\cite{rao2021global}, AFNO\cite{guibas2021efficient}). We hypothesize that both spectral and multi-headed attention plays a major role. We investigate this hypothesis through this work and observe that indeed combining spectral and multi-headed attention layers provides a better transformer architecture. We thus propose the novel Spectformer architecture for transformers that combines spectral and multi-headed attention layers. We believe that the resulting representation allows the transformer to capture the feature representation appropriately and it yields improved performance over other transformer representations. For instance, it improves the top-1 accuracy by 2\% on ImageNet compared to both GFNet-H and LiT. SpectFormer-S reaches 84.25\% top-1 accuracy on ImageNet-1K (state of the art for small version). Further, Spectformer-L achieves 85.7\% that is the state of the art for the comparable base version of the transformers. We further ensure that we obtain reasonable results in other scenarios such as transfer learning on standard datasets such as CIFAR-10, CIFAR-100, Oxford-IIIT-flower, and Standford Car datasets. We then investigate its use in downstream tasks such of object detection and instance segmentation on the MS-COCO dataset and observe that Spectformer shows consistent performance that is comparable to the best backbones and can be further optimized and improved. Hence, we believe that combined spectral and attention layers are what are needed for vision transformers.

翻译：摘要：视觉Transformer已被成功应用于图像识别任务。现有方法要么采用与文本模型原始工作类似的多头自注意力机制（ViT \cite{dosovitskiy2020image}、DeIT \cite{touvron2021training}），要么近期转向基于谱层的方法（Fnet\cite{lee2021fnet}、GFNet\cite{rao2021global}、AFNO\cite{guibas2021efficient}）。我们假设谱方法与多头注意力机制均发挥关键作用，并通过本工作验证该假设，观察到两者结合确实能构建更优的Transformer架构。据此我们提出新型Spectformer架构，融合谱层与多头注意力层。我们认为这种组合表征能使Transformer更有效地捕获特征表示，并相比其他Transformer表征取得性能提升。例如，在ImageNet上，相较于GFNet-H和LiT，该方法将top-1准确率提升2%。SpectFormer-S在ImageNet-1K上达到84.25%的top-1准确率（小型版本最优），而Spectformer-L达到85.7%的top-1准确率（可比较基础版本最优）。此外，我们在CIFAR-10、CIFAR-100、Oxford-IIIT-flower和Stanford Cars等标准数据集的迁移学习实验中验证了其合理性能。进一步在MS-COCO数据集的目标检测与实例分割下游任务中，我们发现Spectformer展现了与最优骨干网络相媲美的稳定性能，并具备进一步优化的潜力。因此我们相信，融合谱层与注意力层正是视觉Transformer所需要的。