This work investigates the computational expressivity of language models (LMs) based on recurrent neural networks (RNNs). Siegelmann and Sontag (1992) famously showed that RNNs with rational weights and hidden states and unbounded computation time are Turing complete. However, LMs define weightings over strings in addition to just (unweighted) language membership and the analysis of the computational power of RNN LMs (RLMs) should reflect this. We extend the Turing completeness result to the probabilistic case, showing how a rationally weighted RLM with unbounded computation time can simulate any deterministic probabilistic Turing machine (PTM) with rationally weighted transitions. Since, in practice, RLMs work in real-time, processing a symbol at every time step, we treat the above result as an upper bound on the expressivity of RLMs. We also provide a lower bound by showing that under the restriction to real-time computation, such models can simulate deterministic real-time rational PTMs.

翻译：本研究探究了基于循环神经网络（RNN）的语言模型（LM）的计算表达能力。Siegelmann 与 Sontag（1992）曾著名地证明，具有有理权重与隐藏状态且计算时间无界的RNN具备图灵完备性。然而，语言模型除了（无权重）语言成员判定外，还定义了字符串上的权重分布，因此对RNN语言模型（RLM）计算能力的分析应体现这一特性。我们将图灵完备性结论扩展到概率情形，展示了具有有理权重且计算时间无界的RLM如何模拟任意具有有理权重转移的确定性概率图灵机（PTM）。鉴于实际应用中RLM以实时方式运行（即每个时间步处理一个符号），我们将上述结论视为RLM表达力的上界。此外，我们通过证明在实时计算约束下此类模型可以模拟确定性实时有理PTM，为其提供了下界。