The community is increasingly exploring linear RNNs (LRNNs) as language models, motivated by their expressive power and parallelizability. While prior work establishes the expressivity benefits of LRNNs over transformers, it is unclear what makes LRNNs -- but not traditional, nonlinear RNNs -- as easy to parallelize in practice as transformers. We answer this question by providing a tight connection between types of RNNs and standard complexity classes. We show that LRNNs can be viewed as log-depth (bounded fan-in) arithmetic circuits, which represents only a slight depth overhead relative to log-depth boolean circuits that transformers admit. Furthermore, we show that nonlinear RNNs can solve $\mathsf{L}$-complete problems (and even $\mathsf{P}$-complete ones, under polynomial precision), revealing a fundamental barrier to parallelizing them as efficiently as transformers. Our theory also identifies fine-grained expressivity differences between recent popular LRNN variants: permutation-diagonal LRNNs are $\mathsf{NC}^1$-complete whereas diagonal-plus-low-rank LRNNs are more expressive ($\mathsf{PNC}^1$-complete). We provide further insight by associating each type of RNN with a corresponding automata-theoretic model that it can simulate. Together, our results reveal fundamental tradeoffs between nonlinear RNNs and different variants of LRNNs, providing a foundation for designing LLM architectures that achieve an optimal balance between expressivity and parallelism.

翻译：学界正日益将线性循环神经网络（LRNN）作为语言模型进行探索，其动因在于这类模型的表达能力与可并行化特性。尽管已有工作确立了LRNN相较Transformer在表达能力上的优势，但为何LRNN（而非传统非线性RNN）在实践中能像Transformer一样易于并行化，这一问题尚待阐明。我们通过建立RNN类型与标准复杂度类之间的紧密关联对此作出解答：研究表明LRNN可视为对数深度（有界扇入）算术电路，相比Transformer所允许的对数深度布尔电路，其深度开销仅略有增加。此外，我们证明非线性RNN能够求解$\mathsf{L}$-完全问题（甚至在多项式精度下求解$\mathsf{P}$-完全问题），这揭示了它们在并行化效率上与Transformer存在根本性障碍。我们的理论还识别出近期流行的LRNN变体之间的细粒度表达差异：置换对角LRNN属于$\mathsf{NC}^1$-完全类，而对角加低秩LRNN具有更强的表达能力（$\mathsf{PNC}^1$-完全类）。通过将每类RNN与相应的自动机理论模型建立关联，我们进一步提供了理论洞见。综合而言，我们的研究结果揭示了非线性RNN与不同LRNN变体之间的根本性权衡，为设计在表达性与并行性之间实现最优平衡的大语言模型架构奠定了基础。