Transformer-based language models utilize the attention mechanism for substantial performance improvements in almost all natural language processing (NLP) tasks. Similar attention structures are also extensively studied in several other areas. Although the attention mechanism enhances the model performances significantly, its quadratic complexity prevents efficient processing of long sequences. Recent works focused on eliminating the disadvantages of computational inefficiency and showed that transformer-based models can still reach competitive results without the attention layer. A pioneering study proposed the FNet, which replaces the attention layer with the Fourier Transform (FT) in the transformer encoder architecture. FNet achieves competitive performances concerning the original transformer encoder model while accelerating training process by removing the computational burden of the attention mechanism. However, the FNet model ignores essential properties of the FT from the classical signal processing that can be leveraged to increase model efficiency further. We propose different methods to deploy FT efficiently in transformer encoder models. Our proposed architectures have smaller number of model parameters, shorter training times, less memory usage, and some additional performance improvements. We demonstrate these improvements through extensive experiments on common benchmarks.

翻译：基于Transformer的语言模型利用注意力机制在几乎所有自然语言处理任务中实现了显著的性能提升。类似的注意力结构在其他多个领域也得到了广泛研究。尽管注意力机制显著增强了模型性能，但其二次复杂度阻碍了长序列的高效处理。近期研究聚焦于消除计算效率低下的缺陷，并表明即使没有注意力层，基于Transformer的模型仍能达到具有竞争力的结果。一项开创性研究提出了FNet，该方法在Transformer编码器架构中用傅里叶变换替代了注意力层。FNet通过消除注意力机制的计算负担加速训练过程，同时达到了与原始Transformer编码器模型相媲美的性能。然而，FNet模型忽略了经典信号处理中FT的关键特性——这些特性可被进一步利用以提升模型效率。我们提出了在Transformer编码器模型中高效部署FT的不同方法。所提出的架构具有更少的模型参数、更短的训练时间、更小的内存占用以及额外的性能提升。通过在通用基准上的大量实验，我们验证了这些改进效果。