Sequential recommendation (SR) models based on Transformers have achieved remarkable successes. The self-attention mechanism of Transformers for computer vision and natural language processing suffers from the oversmoothing problem, i.e., hidden representations becoming similar to tokens. In the SR domain, we, for the first time, show that the same problem occurs. We present pioneering investigations that reveal the low-pass filtering nature of self-attention in the SR, which causes oversmoothing. To this end, we propose a novel method called $\textbf{B}$eyond $\textbf{S}$elf-$\textbf{A}$ttention for Sequential $\textbf{Rec}$ommendation (BSARec), which leverages the Fourier transform to i) inject an inductive bias by considering fine-grained sequential patterns and ii) integrate low and high-frequency information to mitigate oversmoothing. Our discovery shows significant advancements in the SR domain and is expected to bridge the gap for existing Transformer-based SR models. We test our proposed approach through extensive experiments on 6 benchmark datasets. The experimental results demonstrate that our model outperforms 7 baseline methods in terms of recommendation performance. Our code is available at https://github.com/yehjin-shin/BSARec.

翻译：基于Transformer的序列推荐（SR）模型已取得显著成功。用于计算机视觉和自然语言处理的Transformer自注意力机制存在过平滑问题，即隐表示趋于与令牌相似。在SR领域，我们首次证明该问题同样存在。我们通过开创性研究揭示了SR中自注意力的低通滤波本质，正是这一特性导致了过平滑。为此，我们提出了一种名为**B**eyond **S**elf-**A**ttention for Sequential **Rec**ommendation（BSARec）的新方法，该方法利用傅里叶变换实现：（i）通过考虑细粒度序列模式注入归纳偏置；（ii）整合低高频信息以缓解过平滑。我们的发现标志着SR领域的重大进展，有望弥合现有基于Transformer的SR模型之间的鸿沟。我们在6个基准数据集上通过大量实验验证了所提方法，实验结果表明，我们的模型在推荐性能上优于7种基线方法。代码已开源至https://github.com/yehjin-shin/BSARec。