In recent studies, linear recurrent neural networks (LRNNs) have achieved Transformer-level performance in natural language modeling and long-range modeling while offering rapid parallel training and constant inference costs. With the resurged interest in LRNNs, we study whether they can learn the hidden rules in training sequences, such as the grammatical structures of regular language. We theoretically analyze some existing LRNNs and discover their limitations on regular language. Motivated by the analysis, we propose a new LRNN equipped with a block-diagonal and input-dependent transition matrix. Experiments suggest that the proposed model is the only LRNN that can perform length extrapolation on regular language tasks such as Sum, Even Pair, and Modular Arithmetic.

翻译：在最近的研究中，线性循环神经网络（LRNNs）在自然语言建模和长程建模任务上达到了Transformer级别的性能，同时保持了快速的并行训练和恒定的推理成本。随着LRNNs重新引起关注，我们研究了它们能否学习训练序列中的隐藏规则，例如正则语言的语法结构。我们从理论上分析了现有的一些LRNNs，并发现了它们在正则语言处理上的局限性。受此分析启发，我们提出了一种新的LRNN，配备了一个块对角且输入依赖的转移矩阵。实验表明，所提出的模型是唯一能在正则语言任务（如求和、偶数对和模算术）上执行长度外推的LRNN。