Transformer and its variants are fundamental neural architectures in deep learning. Recent works show that learning attention in the Fourier space can improve the long sequence learning capability of Transformers. We argue that wavelet transform shall be a better choice because it captures both position and frequency information with a linear time complexity. Therefore, in this paper, we systematically study the synergy between wavelet transform and Transformers. Specifically, we focus on a new paradigm WISE, which replaces the attention in Transformers by (1) applying forward wavelet transform to project the input sequences to multi-resolution bases, (2) conducting non-linear transformations in the wavelet coefficient space, and (3) reconstructing the representation in input space via backward wavelet transform. Extensive experiments on the Long Range Arena benchmark demonstrate that learning attention in the wavelet space using either fixed or adaptive wavelets can consistently improve Transformer's performance and also significantly outperform Fourier-based methods.

翻译：Transformer及其变体是深度学习中的基础神经架构。近期研究表明，在傅里叶空间中学习注意力机制能够提升Transformer的长序列学习能力。我们论证小波变换应是更优选择，因其能以线性时间复杂度同时捕获位置与频率信息。为此，本文系统研究了小波变换与Transformer之间的协同效应。具体而言，我们聚焦于一种新范式WISE，其通过以下方式替代Transformer中的注意力机制：(1) 应用前向小波变换将输入序列投影到多分辨率基上，(2) 在小波系数空间中进行非线性变换，以及(3) 通过反向小波变换在输入空间中重建表征。在Long Range Arena基准上的大量实验表明，使用固定或自适应小波在小波空间中学习注意力机制，均能持续提升Transformer的性能，并显著优于基于傅里叶的方法。