Studying language models (LMs) in terms of well-understood formalisms allows us to precisely characterize their abilities and limitations. Previous work has investigated the representational capacity of recurrent neural network (RNN) LMs in terms of their capacity to recognize unweighted formal languages. However, LMs do not describe unweighted formal languages -- rather, they define probability distributions over strings. In this work, we study what classes of such probability distributions RNN LMs can represent, which allows us to make more direct statements about their capabilities. We show that simple RNNs are equivalent to a subclass of probabilistic finite-state automata, and can thus model a strict subset of probability distributions expressible by finite-state models. Furthermore, we study the space complexity of representing finite-state LMs with RNNs. We show that, to represent an arbitrary deterministic finite-state LM with $N$ states over an alphabet $\Sigma$, an RNN requires $\Omega\left(N |\Sigma|\right)$ neurons. These results present a first step towards characterizing the classes of distributions RNN LMs can represent and thus help us understand their capabilities and limitations.

翻译：从易于理解的规范形式角度研究语言模型（LMs），能够精确刻画其能力与局限性。已有工作探究了递归神经网络（RNN）语言模型在识别无权重形式语言方面的表征能力。然而，语言模型并非描述无权重形式语言，而是定义字符串上的概率分布。本文研究RNN语言模型所能表征的概率分布类别，从而更直接地评判其能力。我们证明：简单RNN等价于概率有限状态自动机的一个子类，因此仅能建模有限状态模型中可表达概率分布的严格子集。此外，我们研究了用RNN表征有限状态语言模型的空间复杂度，证明：要表征一个具有$N$个状态、字母表为$\Sigma$的任意确定性有限状态语言模型，RNN需要$\Omega\left(N |\Sigma|\right)$个神经元。这些结果标志着刻画RNN语言模型可表征分布类别的初步探索，有助于理解其能力与局限性。