Recurrent and deep neural networks (RNNs/DNNs) are cornerstone architectures in machine learning. Remarkably, RNNs differ from DNNs only by weight sharing, as can be shown through unrolling in time. How does this structural similarity fit with the distinct functional properties these networks exhibit? To address this question, we here develop a unified mean-field theory for RNNs and DNNs in terms of representational kernels, describing fully trained networks in the feature learning ($μ$P) regime. This theory casts training as Bayesian inference over sequences and patterns, directly revealing the functional implications induced by the RNNs' weight sharing. In DNN-typical tasks, we identify a phase transition when the learning signal overcomes the noise due to randomness in the weights: below this threshold, RNNs and DNNs behave identically; above it, only RNNs develop correlated representations across timesteps. For sequential tasks, the RNNs' weight sharing furthermore induces an inductive bias that aids generalization by interpolating unsupervised time steps. Overall, our theory offers a way to connect architectural structure to functional biases.

翻译：循环神经网络与深度神经网络（RNNs/DNNs）是机器学习中的核心架构。值得注意的是，RNN与DNN仅通过权重共享相区别，这可通过时间展开予以证明。这种结构相似性如何与两类网络表现出的不同功能特性相协调？为回答此问题，本文基于表征核为RNN与DNN建立了统一平均场理论，该理论在特征学习（μP）机制下描述完全训练后的网络。该理论将训练过程视为对序列与模式的贝叶斯推断，直接揭示了RNN权重共享所引发的功能内涵。在DNN典型任务中，我们识别出当学习信号超越权重随机性产生的噪声时出现的相变：低于该阈值时，RNN与DNN表现一致；高于该阈值时，仅RNN能跨时间步发展出相关表征。对于序列任务，RNN的权重共享进一步产生了一种归纳偏置，通过插值无监督时间步来促进泛化能力。总体而言，我们的理论提供了一种将架构结构与功能偏置相联系的途径。