Recent advances in self-attention and pure multi-layer perceptrons (MLP) models for vision have shown great potential in achieving promising performance with fewer inductive biases. These models are generally based on learning interaction among spatial locations from raw data. The complexity of self-attention and MLP grows quadratically as the image size increases, which makes these models hard to scale up when high-resolution features are required. In this paper, we present the Global Filter Network (GFNet), a conceptually simple yet computationally efficient architecture, that learns long-term spatial dependencies in the frequency domain with log-linear complexity. Our architecture replaces the self-attention layer in vision transformers with three key operations: a 2D discrete Fourier transform, an element-wise multiplication between frequency-domain features and learnable global filters, and a 2D inverse Fourier transform. We exhibit favorable accuracy/complexity trade-offs of our models on both ImageNet and downstream tasks. Our results demonstrate that GFNet can be a very competitive alternative to transformer-style models and CNNs in efficiency, generalization ability and robustness. Code is available at https://github.com/raoyongming/GFNet

翻译：自我注意和纯多层视觉模型(MLP)最近的进展显示,在以较少的感化偏差实现有希望的性能方面具有巨大潜力。这些模型一般以原始数据的空间位置之间的学习互动为基础。自我注意和MLP的复杂性随着图像大小的增加而增长四边形,这使得这些模型在需要高分辨率特征时难以扩大。在本文中,我们介绍了全球过滤网络(GFNet),这是一个概念简单、但计算效率高的结构,它学习了以对线性复杂度在频率区域的长期空间依赖性。我们的建筑用三种关键操作取代了视觉变异器中的自留层:2D离散的Fourier变换,频率特性和可学习的全球过滤器之间的元素性倍增,以及2D在需要高分辨率特性时难以扩大。我们展示了在图像网和下游任务上的模型的准确性/兼容性交换。我们的结果显示,GFNet可以非常有竞争力,替代变异式模型和CNN在效率、通用能力和强性网络/GFMMMAR/GMS。可提供的代码。