Mathematical methods are developed to characterize the asymptotics of recurrent neural networks (RNN) as the number of hidden units, data samples in the sequence, hidden state updates, and training steps simultaneously grow to infinity. In the case of an RNN with a simplified weight matrix, we prove the convergence of the RNN to the solution of an infinite-dimensional ODE coupled with the fixed point of a random algebraic equation. The analysis requires addressing several challenges which are unique to RNNs. In typical mean-field applications (e.g., feedforward neural networks), discrete updates are of magnitude $\mathcal{O}(\frac{1}{N})$ and the number of updates is $\mathcal{O}(N)$. Therefore, the system can be represented as an Euler approximation of an appropriate ODE/PDE, which it will converge to as $N \rightarrow \infty$. However, the RNN hidden layer updates are $\mathcal{O}(1)$. Therefore, RNNs cannot be represented as a discretization of an ODE/PDE and standard mean-field techniques cannot be applied. Instead, we develop a fixed point analysis for the evolution of the RNN memory states, with convergence estimates in terms of the number of update steps and the number of hidden units. The RNN hidden layer is studied as a function in a Sobolev space, whose evolution is governed by the data sequence (a Markov chain), the parameter updates, and its dependence on the RNN hidden layer at the previous time step. Due to the strong correlation between updates, a Poisson equation must be used to bound the fluctuations of the RNN around its limit equation. These mathematical methods give rise to the neural tangent kernel (NTK) limits for RNNs trained on data sequences as the number of data samples and size of the neural network grow to infinity.

翻译：本文发展了数学方法，用以刻画当隐藏单元数、序列中的数据样本数、隐藏状态更新次数以及训练步数同时趋于无穷大时，循环神经网络（RNN）的渐近行为。针对具有简化权重矩阵的RNN，我们证明了该RNN收敛于一个无限维常微分方程（ODE）的解，该ODE与一个随机代数方程的不动点相关联。分析过程需要应对RNN特有的若干挑战。在典型的平均场应用（如前馈神经网络）中，离散更新的量级为$\mathcal{O}(\frac{1}{N})$，更新次数为$\mathcal{O}(N)$。因此，系统可表示为相应ODE/PDE的欧拉近似，且当$N \rightarrow \infty$时系统将收敛于此ODE/PDE。然而，RNN隐藏层更新的量级为$\mathcal{O}(1)$。故RNN无法表示为ODE/PDE的离散化形式，标准平均场技术也无法适用。为此，我们针对RNN记忆状态的演化发展了不动点分析，并给出了关于更新步数和隐藏单元数的收敛估计。我们将RNN隐藏层视为Sobolev空间中的函数，其演化由数据序列（马尔可夫链）、参数更新及其对前一时间步RNN隐藏层的依赖性共同决定。由于更新之间存在强相关性，必须使用泊松方程来约束RNN相对于其极限方程的波动。当数据样本数与神经网络规模趋于无穷大时，这些数学方法导出了在数据序列上训练的RNN的神经正切核（NTK）极限。