Studying language models (LMs) in terms of well-understood formalisms allows us to precisely characterize their abilities and limitations. Previous work has investigated the representational capacity of recurrent neural network (RNN) LMs in terms of their capacity to recognize unweighted formal languages. However, LMs do not describe unweighted formal languages -- rather, they define \emph{probability distributions} over strings. In this work, we study what classes of such probability distributions RNN LMs can represent, which allows us to make more direct statements about their capabilities. We show that simple RNNs are equivalent to a subclass of probabilistic finite-state automata, and can thus model a strict subset of probability distributions expressible by finite-state models. Furthermore, we study the space complexity of representing finite-state LMs with RNNs. We show that, to represent an arbitrary deterministic finite-state LM with $N$ states over an alphabet $\alphabet$, an RNN requires $\Omega\left(N |\Sigma|\right)$ neurons. These results present a first step towards characterizing the classes of distributions RNN LMs can represent and thus help us understand their capabilities and limitations.

翻译：以理解充分的规范形式研究语言模型，使我们能够精确刻画其能力与局限性。先前的研究从识别未加权形式语言的能力角度，探讨了循环神经网络语言模型的表示容量。然而，语言模型描述的是字符串上的*概率分布*，而非未加权形式语言。本文研究循环神经网络语言模型能够表示的概率分布类别，从而对其能力做出更直接的论断。我们证明，简单循环网络等价于概率有限状态自动机的一个子类，因此仅能建模有限状态模型可表达概率分布的严格子集。此外，我们研究了用循环网络表示有限状态语言模型的空间复杂度。结果表明，要表示一个具有$N$个状态、字母表为$\alphabet$的确定性有限状态语言模型，循环网络需要$\Omega\left(N |\Sigma|\right)$个神经元。这些结果迈出了刻画循环网络语言模型可表示分布类别的第一步，有助于理解其能力与局限性。