Multi-head-self-attention (MHSA)-equipped models have achieved notable performance in computer vision. Their computational complexity is proportional to quadratic numbers of pixels in input feature maps, resulting in slow processing, especially when dealing with high-resolution images. New types of token-mixer are proposed as an alternative to MHSA to circumvent this problem: an FFT-based token-mixer, similar to MHSA in global operation but with lower computational complexity. However, despite its attractive properties, the FFT-based token-mixer has not been carefully examined in terms of its compatibility with the rapidly evolving MetaFormer architecture. Here, we propose a novel token-mixer called dynamic filter and DFFormer and CDFFormer, image recognition models using dynamic filters to close the gaps above. CDFFormer achieved a Top-1 accuracy of 85.0%, close to the hybrid architecture with convolution and MHSA. Other wide-ranging experiments and analysis, including object detection and semantic segmentation, demonstrate that they are competitive with state-of-the-art architectures; Their throughput and memory efficiency when dealing with high-resolution image recognition is convolution and MHSA, not much different from ConvFormer, and far superior to CAFormer. Our results indicate that the dynamic filter is one of the token-mixer options that should be seriously considered. The code is available at https://github.com/okojoalg/dfformer

翻译：配备多头自注意力（MHSA）的模型在计算机视觉领域取得了显著性能。其计算复杂度与输入特征图中的像素数量呈二次关系，导致处理速度缓慢，尤其是在处理高分辨率图像时。为解决此问题，人们提出了新型令牌混合器作为MHSA的替代方案：基于FFT的令牌混合器，其在全局操作上与MHSA类似，但计算复杂度更低。然而，尽管基于FFT的令牌混合器具有吸引力，但其与快速发展的MetaFormer架构的兼容性尚未得到仔细检验。本文提出了一种名为动态滤波器的新型令牌混合器，以及DFFormer和CDFFormer图像识别模型，利用动态滤波器填补上述空白。CDFFormer实现了85.0%的Top-1准确率，接近采用卷积与MHSA的混合架构。涵盖目标检测和语义分割的广泛实验与分析表明，它们与最先进架构相比具有竞争力；其在高分辨率图像识别任务中的吞吐量和内存效率与ConvFormer相当，但远优于CAFormer。我们的结果表明，动态滤波器是值得认真考虑的令牌混合器选项之一。代码已开源在https://github.com/okojoalg/dfformer