The recent success of multiple neural architectures like CNNs, Transformers, and MLP-Mixers motivated us to look for similarities and differences between them. We found that these architectures can be interpreted through the lens of a general concept of dimension mixing. Research on coupling flows and the butterfly transform shows that partial and hierarchical signal mixing schemes are sufficient for efficient and expressive function approximation. In this work, we study group-wise sparse, non-linear, multi-layered and learnable mixing schemes of inputs and find that they are complementary to many standard neural architectures. Following our observations and drawing inspiration from the Fast Fourier Transform, we generalize Butterfly Structure to use non-linear mixer function allowing for MLP as mixing function called Butterfly MLP. We were also able to mix along sequence dimension for Transformer-based architectures called Butterfly Attention. Experiments on CIFAR and LRA datasets demonstrate that the proposed Non-Linear Butterfly Mixers are efficient and scale well when the host architectures are used as mixing function. Additionally, we propose Patch-Only MLP-Mixer for processing spatial 2D signals demonstrating a different dimension mixing strategy.

翻译：近年来，卷积神经网络、Transformer和MLP-Mixer等多种神经架构的成功，促使我们探寻它们之间的共性与差异。我们发现，这些架构可通过“维度混合”这一通用概念进行解析。关于耦合流与蝴蝶变换的研究表明，部分分层信号混合方案足以实现高效且具表现力的函数逼近。本研究探讨了分组稀疏、非线性、多层可学习的输入混合方案，并发现它们与多种标准神经架构具有互补性。基于观察结果并受快速傅里叶变换启发，我们将蝴蝶结构泛化为允许使用MLP作为混合函数的非线性混合器（称为Butterfly MLP），同时还实现了沿序列维度混合的Transformer变体（称为Butterfly Attention）。在CIFAR和LRA数据集上的实验表明，所提出的非线性蝴蝶混合器在宿主架构作为混合函数时兼具高效性与可扩展性。此外，我们提出仅处理二维空间信号的Patch-Only MLP-Mixer，展示了一种不同维度的混合策略。