Residual connections have been proposed as architecture-based inductive bias to mitigate the problem of exploding and vanishing gradients and increase task performance in both feed-forward and recurrent networks (RNNs) when trained with the backpropagation algorithm. Yet, little is known about how residual connections in RNNs influence their dynamics and fading memory properties. Here, we introduce weakly coupled residual recurrent networks (WCRNNs) in which residual connections result in well-defined Lyapunov exponents and allow for studying properties of fading memory. We investigate how the residual connections of WCRNNs influence their performance, network dynamics, and memory properties on a set of benchmark tasks. We show that several distinct forms of residual connections yield effective inductive biases that result in increased network expressivity. In particular, residual connections that (i) result in network dynamics at the proximity of the edge of chaos, (ii) allow networks to capitalize on characteristic spectral properties of the data, and (iii) result in heterogeneous memory properties are shown to increase practical expressivity. In addition, we demonstrate how our results can be extended to non-linear residuals and introduce a weakly coupled residual initialization scheme that can be used for Elman RNNs

翻译：残差连接已被提出作为一种基于架构的归纳偏置，用于缓解在使用反向传播算法训练前馈网络和循环网络（RNN）时出现的梯度爆炸和梯度消失问题，并提升任务性能。然而，关于残差连接如何影响RNN的动态特性及残差记忆性质，目前仍知之甚少。本文引入了弱耦合残差循环网络（WCRNNs），其中残差连接产生了明确定义的李雅普诺夫指数，并允许研究残差记忆的性质。我们在一组基准任务上探究了WCRNN的残差连接如何影响其性能、网络动态及记忆特性。我们表明，多种不同形式的残差连接能产生有效的归纳偏置，从而提升网络表达力。特别地，残差连接若能够：(i) 使网络动态接近混沌边缘；(ii) 允许网络利用数据的特征谱性质；以及(iii) 产生异质性的记忆特性，则能有效提升实际表达力。此外，我们演示了如何将结果扩展到非线性残差，并引入了一种可用于Elman RNN的弱耦合残差初始化方案。