Recent vision transformers, large-kernel CNNs and MLPs have attained remarkable successes in broad vision tasks thanks to their effective information fusion in the global scope. However, their efficient deployments, especially on mobile devices, still suffer from noteworthy challenges due to the heavy computational costs of self-attention mechanisms, large kernels, or fully connected layers. In this work, we apply conventional convolution theorem to deep learning for addressing this and reveal that adaptive frequency filters can serve as efficient global token mixers. With this insight, we propose Adaptive Frequency Filtering (AFF) token mixer. This neural operator transfers a latent representation to the frequency domain via a Fourier transform and performs semantic-adaptive frequency filtering via an elementwise multiplication, which mathematically equals to a token mixing operation in the original latent space with a dynamic convolution kernel as large as the spatial resolution of this latent representation. We take AFF token mixers as primary neural operators to build a lightweight neural network, dubbed AFFNet. Extensive experiments demonstrate the effectiveness of our proposed AFF token mixer and show that AFFNet achieve superior accuracy and efficiency trade-offs compared to other lightweight network designs on broad visual tasks, including visual recognition and dense prediction tasks.

翻译：近期，视觉Transformer、大核CNN和MLP凭借其在全局范围内的有效信息融合，已在广泛视觉任务中取得了显著成功。然而，其高效部署（尤其在移动设备上）仍面临重大挑战，原因在于自注意力机制、大核或全连接层带来的高昂计算成本。在本工作中，我们将传统卷积定理应用于深度学习以解决此问题，并揭示自适应频率滤波器可作为高效的全局令牌混合器。基于这一见解，我们提出自适应频率滤波（AFF）令牌混合器。该神经算子通过傅里叶变换将潜在表示转换至频域，并通过逐元素乘法执行语义自适应的频率滤波，其在数学上等价于在原始潜在空间中利用与空间分辨率相同大小的动态卷积核进行令牌混合操作。我们将AFF令牌混合器作为主要神经算子，构建轻量级神经网络AFFNet。大量实验表明，我们提出的AFF令牌混合器具有有效性，且AFFNet在包括视觉识别和密集预测任务在内的广泛视觉任务中，相较于其他轻量级网络设计实现了更优的精度与效率权衡。