Structured state-space models (SSMs) such as S4, stemming from the seminal work of Gu et al., are gaining popularity as effective approaches for modeling sequential data. Deep SSMs demonstrate outstanding performance across a diverse set of domains, at a reduced training and inference cost compared to attention-based transformers. Recent developments show that if the linear recurrence powering SSMs allows for multiplicative interactions between inputs and hidden states (e.g. GateLoop, Mamba, GLA), then the resulting architecture can surpass in both in accuracy and efficiency attention-powered foundation models trained on text, at scales of billion parameters. In this paper, we give theoretical grounding to this recent finding using tools from Rough Path Theory: we show that when random linear recurrences are equipped with simple input-controlled transitions (selectivity mechanism), then the hidden state is provably a low-dimensional projection of a powerful mathematical object called the signature of the input -- capturing non-linear interactions between tokens at distinct timescales. Our theory not only motivates the success of modern selective state-space models such as Mamba but also provides a solid framework to understand the expressive power of future SSM variants.

翻译：结构化状态空间模型（SSMs），如源自Gu等人开创性工作的S4，正作为序列数据建模的有效方法日益受到关注。深度SSMs在降低训练和推理成本的同时，在多个领域展现出卓越性能，其效率优于基于注意力的Transformer。最新进展表明，若驱动SSMs的线性循环允许输入与隐藏状态间存在乘法交互（例如GateLoop、Mamba、GLA），则由此产生的架构在准确性及效率上均能超越在文本上训练、参数规模达十亿级的基于注意力的基础模型。本文利用粗路径理论工具为这一最新发现提供理论依据：我们证明，当随机线性循环配备简单的输入控制转换（选择性机制）时，隐藏状态可被证明是名为输入签名这一强大数学对象的低维投影——该对象能捕捉不同时间尺度上词元间的非线性交互。我们的理论不仅阐释了Mamba等现代选择性状态空间模型成功的原因，更为理解未来SSM变体的表达能力提供了坚实的框架。