Linear recurrent networks (LRNNs) and linear state space models (SSMs) promise computational and memory efficiency on long-sequence modeling tasks, yet their diagonal state transitions limit expressivity. Dense and nonlinear architectures (e.g., LSTMs) on the other hand are provably more expressive, but computationally costly. Here, we explore how expressivity in LRNNs can be increased via richer state mixing across time and channels while maintaining competitive efficiency. Specifically, we introduce two structured LRNN architectures: (i) Higher-order Linear Recurrent Units (H-LRU), which generalize first-order recurrence to higher order, mixing multiple past states, and (ii) Block-Diagonal LRUs (BD-LRU), which enable dense intra-block channel mixing. Per-channel (H-LRU) or per-row (BD-LRU) L1-normalization of selective gates stabilizes training and allows for scaling window/block sizes. A parallel-scan implementation of the proposed architectures keeps the throughput competitive with diagonal LRNNs for moderate orders (H-LRU) and block sizes (BD-LRU). In synthetic sequence modeling tasks, the performance of BD-LRU matches or exceeds those of linear SSMs (Mamba), low-rank LRNNs (DeltaNet) and LSTM baselines, while H-LRU is found to be the most parameter-efficient in compression task. In both synthetic sequence modeling and language modeling, our results indicate that the structure of state mixing rather than width alone shapes expressivity of LRNNs, offering a practical route to closing the efficiency-expressivity gap in linear sequence models.

翻译：线性循环网络（LRNNs）与线性状态空间模型（SSMs）在长序列建模任务中具有计算与内存效率优势，但其对角状态转移限制了表达能力。另一方面，稠密与非线性的架构（如LSTM）被证明更具表达能力，但计算成本高昂。本文探讨了如何在保持竞争性效率的同时，通过跨时间与通道的更丰富状态混合来提升LRNNs的表达能力。具体而言，我们引入了两种结构化LRNN架构：（i）高阶线性循环单元（H-LRU），将一阶递推推广至高阶，混合多个过去状态；（ii）块对角LRU（BD-LRU），实现块内稠密的通道混合。选择性门控的逐通道（H-LRU）或逐行（BD-LRU）L1归一化稳定了训练，并允许扩展窗口/块大小。所提架构的并行扫描实现使其在中等阶数（H-LRU）与块大小（BD-LRU）下，吞吐量与对角LRNNs保持竞争水平。在合成序列建模任务中，BD-LRU的性能匹配或超越了线性SSMs（Mamba）、低秩LRNNs（DeltaNet）及LSTM基线，而H-LRU在压缩任务中被发现是参数效率最高的。在合成序列建模与语言建模中，我们的结果表明，状态混合的结构而非仅宽度决定了LRNNs的表达能力，为弥合线性序列模型的效率-表达能力差距提供了一条实用路径。