This work investigates the computational expressivity of language models (LMs) based on recurrent neural networks (RNNs). Siegelmann and Sontag (1992) famously showed that RNNs with rational weights and hidden states and unbounded computation time are Turing complete. However, LMs define weightings over strings in addition to just (unweighted) language membership and the analysis of the computational power of RNN LMs (RLMs) should reflect this. We extend the Turing completeness result to the probabilistic case, showing how a rationally weighted RLM with unbounded computation time can simulate any deterministic probabilistic Turing machine (PTM) with rationally weighted transitions. Since, in practice, RLMs work in real-time, processing a symbol at every time step, we treat the above result as an upper bound on the expressivity of RLMs. We also provide a lower bound by showing that under the restriction to real-time computation, such models can simulate deterministic real-time rational PTMs.

翻译：本研究探讨了基于循环神经网络的语言模型的计算表达能力。Siegelmann 与 Sontag (1992) 的著名研究表明，具有有理权重与隐藏状态且计算时间不受限制的 RNN 是图灵完备的。然而，语言模型不仅定义了（未加权的）语言成员资格，还定义了字符串上的加权分布，因此对 RNN 语言模型计算能力的分析应反映这一特性。我们将图灵完备性结果扩展至概率情形，证明了具有无限计算时间的有理加权 RLM 如何能够模拟任何具有有理加权转移的确定性概率图灵机。由于在实际应用中，RLM 以实时方式工作，在每个时间步处理一个符号，我们将上述结果视为 RLM 表达能力的一个上界。我们还通过证明在实时计算限制下，此类模型能够模拟确定性实时有理概率图灵机，从而提供了一个下界。