A longstanding challenge for the Machine Learning community is the one of developing models that are capable of processing and learning from very long sequences of data. The outstanding results of Transformers-based networks (e.g., Large Language Models) promotes the idea of parallel attention as the key to succeed in such a challenge, obfuscating the role of classic sequential processing of Recurrent Models. However, in the last few years, researchers who were concerned by the quadratic complexity of self-attention have been proposing a novel wave of neural models, which gets the best from the two worlds, i.e., Transformers and Recurrent Nets. Meanwhile, Deep Space-State Models emerged as robust approaches to function approximation over time, thus opening a new perspective in learning from sequential data, followed by many people in the field and exploited to implement a special class of (linear) Recurrent Neural Networks. This survey is aimed at providing an overview of these trends framed under the unifying umbrella of Recurrence. Moreover, it emphasizes novel research opportunities that become prominent when abandoning the idea of processing long sequences whose length is known-in-advance for the more realistic setting of potentially infinite-length sequences, thus intersecting the field of lifelong-online learning from streamed data.

翻译：机器学习领域长期面临的一项挑战是开发能够处理和学习极长序列数据的模型。基于Transformer的网络（如大型语言模型）的卓越表现，使得并行注意力机制被视为应对该挑战的关键，从而掩盖了循环模型经典序列处理的作用。然而近几年来，受自注意力二次复杂度困扰的研究人员提出了一波新型神经模型，这些模型融合了Transformer和循环网络两大范式的优势。与此同时，深度空间状态模型作为时间域函数逼近的稳健方法应运而生，为序列数据学习开辟了新视角，吸引了众多领域研究者，并被用于实现特殊类别（线性）循环神经网络。本综述旨在以循环性为统一框架，梳理这些研究趋势。此外，本文着重强调了当抛弃"预先知道序列长度"的处理范式，转向更贴近现实场景的潜在无限长序列处理时，将涌现出的新兴研究机遇——这恰好与流式数据的终身在线学习领域形成交叉。