Vision transformers have been applied successfully for image recognition tasks. There have been either multi-headed self-attention based (ViT \cite{dosovitskiy2020image}, DeIT, \cite{touvron2021training}) similar to the original work in textual models or more recently based on spectral layers (Fnet\cite{lee2021fnet}, GFNet\cite{rao2021global}, AFNO\cite{guibas2021efficient}). We hypothesize that both spectral and multi-headed attention plays a major role. We investigate this hypothesis through this work and observe that indeed combining spectral and multi-headed attention layers provides a better transformer architecture. We thus propose the novel Spectformer architecture for transformers that combines spectral and multi-headed attention layers. We believe that the resulting representation allows the transformer to capture the feature representation appropriately and it yields improved performance over other transformer representations. For instance, it improves the top-1 accuracy by 2\% on ImageNet compared to both GFNet-H and LiT. SpectFormer-S reaches 84.25\% top-1 accuracy on ImageNet-1K (state of the art for small version). Further, Spectformer-L achieves 85.7\% that is the state of the art for the comparable base version of the transformers. We further ensure that we obtain reasonable results in other scenarios such as transfer learning on standard datasets such as CIFAR-10, CIFAR-100, Oxford-IIIT-flower, and Standford Car datasets. We then investigate its use in downstream tasks such of object detection and instance segmentation on the MS-COCO dataset and observe that Spectformer shows consistent performance that is comparable to the best backbones and can be further optimized and improved. Hence, we believe that combined spectral and attention layers are what are needed for vision transformers.

翻译：视觉Transformer已成功应用于图像识别任务。现有工作主要分为两类：一类基于多头自注意力机制（ViT\cite{dosovitskiy2020image}、DeIT\cite{touvron2021training}），沿袭了文本模型的原始设计；另一类近期方法则基于谱层（Fnet\cite{lee2021fnet}、GFNet\cite{rao2021global}、AFNO\cite{guibas2021efficient}）。我们假设光谱域与多头注意力机制均发挥着关键作用，并通过本研究验证了这一假设，观察到将光谱层与多头注意力层相结合确实能构建更优的Transformer架构。据此，我们提出创新的Spectformer架构，通过融合光谱层与多头注意力层，使Transformer能够更有效地捕获特征表示，相比其他Transformer表示方法取得性能提升。例如，在ImageNet上，相比GFNet-H和LiT，其Top-1准确率提升2%。SpectFormer-S在ImageNet-1K上达到84.25%的Top-1准确率（小型版本达到当前最优）。此外，Spectformer-L达到85.7%的Top-1准确率，在可比的基础版本Transformer中亦为最优。我们进一步验证了其在迁移学习场景中的有效性（涵盖CIFAR-10、CIFAR-100、Oxford-IIIT-flower及Stanford Car等标准数据集）。基于MS-COCO数据集的目标检测与实例分割下游任务实验表明，Spectformer展现出与最优骨干网络相当的一致性能，并具备进一步优化提升的潜力。因此，我们认为视觉Transformer需要融合光谱层与注意力层。