Free-running Recurrent Neural Networks (RNNs), especially probabilistic models, generate an ongoing information flux that can be quantified with the mutual information $I\left[\vec{x}(t),\vec{x}(t\!+\!1)\right]$ between subsequent system states $\vec{x}$. Although, former studies have shown that $I$ depends on the statistics of the network's connection weights, it is unclear (1) how to maximize $I$ systematically and (2) how to quantify the flux in large systems where computing the mutual information becomes intractable. Here, we address these questions using Boltzmann machines as model systems. We find that in networks with moderately strong connections, the mutual information $I$ is approximately a monotonic transformation of the root-mean-square averaged Pearson correlations between neuron-pairs, a quantity that can be efficiently computed even in large systems. Furthermore, evolutionary maximization of $I\left[\vec{x}(t),\vec{x}(t\!+\!1)\right]$ reveals a general design principle for the weight matrices enabling the systematic construction of systems with a high spontaneous information flux. Finally, we simultaneously maximize information flux and the mean period length of cyclic attractors in the state space of these dynamical networks. Our results are potentially useful for the construction of RNNs that serve as short-time memories or pattern generators.

翻译：自由运行的递归神经网络（RNN），特别是概率模型，会产生持续的信息通量，该通量可通过系统连续状态 $\vec{x}(t)$ 与 $\vec{x}(t+1)$ 之间的互信息 $I\left[\vec{x}(t),\vec{x}(t\!+\!1)\right]$ 进行量化。尽管已有研究表明 $I$ 取决于网络连接权重的统计特性，但仍不清楚：(1) 如何系统性地最大化 $I$；(2) 如何在大型系统中量化该通量——因为此时互信息的计算变得难以处理。本文以玻尔兹曼机为模型系统，针对上述问题展开研究。我们发现，在连接强度适中的网络中，互信息 $I$ 近似为神经元对间皮尔逊相关系数均方根平均值的单调变换，该变量即使在大型系统中也能高效计算。此外，对 $I\left[\vec{x}(t),\vec{x}(t\!+\!1)\right]$ 的进化最大化揭示了权重矩阵的通用设计原则，从而能够系统性地构建具有高自发信息通量的系统。最后，我们同时最大化信息通量以及这些动力学网络状态空间中循环吸引子的平均周期长度。研究结果对于构建用作短时记忆或模式生成器的循环神经网络具有潜在应用价值。