Linear Recurrence has proven to be a powerful tool for modeling long sequences efficiently. In this work, we show that existing models fail to take full advantage of its potential. Motivated by this finding, we develop GateLoop, a foundational sequence model that generalizes linear recurrent models such as S4, S5, LRU and RetNet, by employing data-controlled state transitions. Utilizing this theoretical advance, GateLoop empirically outperforms existing models for auto-regressive language modeling. Our method comes with a low-cost $O(l)$ recurrent mode and an efficient $O(l \log_{2} l)$ parallel mode making use of highly optimized associative scan implementations. Furthermore, we derive an $O(l^2)$ surrogate attention mode, revealing remarkable implications for Transformer and recently proposed architectures. Specifically, we prove that our approach can be interpreted as providing data-controlled relative-positional information to Attention. While many existing models solely rely on data-controlled cumulative sums for context aggregation, our findings suggest that incorporating data-controlled complex cumulative products may be a crucial step towards more powerful sequence models.

翻译：线性递归已被证明是高效建模长序列的强大工具。本文表明，现有模型未能充分利用其潜力。受此发现启发，我们提出了GateLoop——一种基础序列模型，通过采用数据控制的状态转移，泛化了S4、S5、LRU和RetNet等线性递归模型。借助这一理论突破，GateLoop在自回归语言建模任务上经验性地超越了现有模型。我们的方法包含低成本的$O(l)$递归模式，以及利用高度优化的关联扫描实现的高效$O(l \log_{2} l)$并行模式。此外，我们推导出$O(l^2)$的替代注意力机制，揭示了其对Transformer及近期提出的架构具有重要启示。具体而言，我们证明该方法可被解释为向注意力机制提供数据控制的相对位置信息。尽管现有模型多依赖数据控制的累积和进行上下文聚合，但我们的发现表明，引入数据控制的复数累积乘积可能是迈向更强大序列模型的关键一步。