In Sequential Recommendation Systems (SRSs), Transformer models have demonstrated remarkable performance but face computational and memory cost challenges, especially when modeling long-term user behavior sequences. Due to its quadratic complexity, the dot-product attention mechanism in Transformers becomes expensive for processing long sequences. By approximating the dot-product attention using elaborate mapping functions, linear attention provides a more efficient option with linear complexity. However, existing linear attention methods face three limitations: 1) they often use learnable position encodings, which incur extra computational costs in long-term sequence scenarios, 2) limited by the low-rank deficiency, they may not sufficiently account for user's fine-grained local preferences (short-lived burst of interest), and 3) they try to capture some temporary activities, but often confuse these with stable and long-term interests. This can result in unclear or less effective recommendations. To remedy these drawbacks, we propose a long-term sequential Recommendation model with Gated Rotary Enhanced Linear Attention (RecGRELA). Specifically, we first propose a Rotary-Enhanced Linear Attention (RELA) module to efficiently model long-range dependency within the user's historical information using rotary position encodings. Then, to address the low-rank deficiency of linear attention, we introduce an Adaptive Rank Modulator. It incorporates a rank augmentation branch to explicitly inject local token mixing and a Gated Rank Selector to dynamically balance stable long-term preferences and transient short-term interests. Experimental results on four public benchmark datasets show that our RecGRELA achieves state-of-the-art performance compared with existing SRSs based on Recurrent Neural Networks, Transformer, and Mamba while keeping low memory overhead.

翻译：在序列推荐系统（SRS）中，Transformer模型展现出卓越性能，但在建模用户长期行为序列时面临计算和内存成本挑战。由于其二次复杂度，Transformer中的点积注意力机制在处理长序列时计算开销高昂。通过利用精心设计的映射函数近似点积注意力，线性注意力以线性复杂度提供了更高效的方案。然而，现有线性注意力方法存在三个局限：1）常使用可学习位置编码，在长期序列场景中产生额外计算成本；2）受限于低秩缺陷，可能无法充分捕捉用户细粒度局部偏好（短暂的兴趣爆发）；3）试图捕获某些临时活动，却常与稳定长期兴趣混淆，导致推荐结果模糊或低效。为解决这些问题，我们提出基于门控旋转增强线性注意力的长期序列推荐模型（RecGRELA）。具体而言，我们首先提出旋转增强线性注意力（RELA）模块，利用旋转位置编码高效建模用户历史信息中的长程依赖关系。其次，针对线性注意力的低秩缺陷，引入自适应秩调制器，通过秩增强分支显式注入局部标记混合，并利用门控秩选择器动态平衡稳定的长期偏好与瞬时的短期兴趣。在四个公开基准数据集上的实验结果表明，与基于循环神经网络、Transformer和Mamba的现有SRS相比，RecGRELA在保持低内存开销的同时实现了最先进的性能。