A recurrent neural network (RNN) is a widely used deep-learning network for dealing with sequential data. Imitating a dynamical system, an infinite-width RNN can approximate any open dynamical system in a compact domain. In general, deep networks with bounded widths are more effective than wide networks in practice; however, the universal approximation theorem for deep narrow structures has yet to be extensively studied. In this study, we prove the universality of deep narrow RNNs and show that the upper bound of the minimum width for universality can be independent of the length of the data. Specifically, we show that a deep RNN with ReLU activation can approximate any continuous function or $L^p$ function with the widths $d_x+d_y+2$ and $\max\{d_x+1,d_y\}$, respectively, where the target function maps a finite sequence of vectors in $\mathbb{R}^{d_x}$ to a finite sequence of vectors in $\mathbb{R}^{d_y}$. We also compute the additional width required if the activation function is $\tanh$ or more. In addition, we prove the universality of other recurrent networks, such as bidirectional RNNs. Bridging a multi-layer perceptron and an RNN, our theory and proof technique can be an initial step toward further research on deep RNNs.

翻译：循环神经网络（RNN）是一种广泛用于处理序列数据的深度学习网络。通过模拟动力系统，无限宽度的RNN可以在紧致域内逼近任意开动力系统。通常在实践中，有界宽度的深层网络比宽网络更有效；然而，针对深层窄结构的普适逼近定理尚未得到广泛研究。本研究证明了深层窄RNN的普适性，并表明普适性所需最小宽度的上界可以独立于数据长度。具体而言，我们证明：使用ReLU激活函数的深度RNN能够分别以宽度$d_x+d_y+2$和$\max\{d_x+1,d_y\}$逼近任意连续函数或$L^p$函数，其中目标函数将$\mathbb{R}^{d_x}$中的有限向量序列映射为$\mathbb{R}^{d_y}$中的有限向量序列。我们还计算了当激活函数为$\tanh$或其它函数时所需的额外宽度。此外，我们证明了其他循环网络（如双向RNN）的普适性。通过桥接多层感知机与RNN，我们的理论与证明技术可为深度RNN的后续研究奠定基础。