Numerical evaluations of the memory capacity (MC) of recurrent neural networks reported in the literature often contradict well-established theoretical bounds. In this paper, we study the case of linear echo state networks, for which the total memory capacity has been proven to be equal to the rank of the corresponding Kalman controllability matrix. We shed light on various reasons for the inaccurate numerical estimations of the memory, and we show that these issues, often overlooked in the recent literature, are of an exclusively numerical nature. More explicitly, we prove that when the Krylov structure of the linear MC is ignored, a gap between the theoretical MC and its empirical counterpart is introduced. As a solution, we develop robust numerical approaches by exploiting a result of MC neutrality with respect to the input mask matrix. Simulations show that the memory curves that are recovered using the proposed methods fully agree with the theory.

翻译：文献中报告的对递归神经网络记忆容量（MC）的数值评估常常与成熟的理论界相矛盾。本文研究了线性回声状态网络的情况，对于这类网络，总记忆容量已被证明等于相应卡尔曼可控性矩阵的秩。我们揭示了导致记忆容量数值估计不准确的各种原因，并表明这些在近期文献中常被忽视的问题纯属数值性质。更明确地说，我们证明当忽略线性MC的Krylov结构时，理论MC与其经验值之间会出现差距。作为解决方案，我们通过利用MC相对于输入掩码矩阵的中性结果，开发了稳健的数值方法。仿真表明，使用所提方法恢复的记忆曲线与理论完全吻合。