Recurrent neural networks are used to forecast time series in finance, climate, language, and from many other domains. Reservoir computers are a particularly easily trainable form of recurrent neural network. Recently, a "next-generation" reservoir computer was introduced in which the memory trace involves only a finite number of previous symbols. We explore the inherent limitations of finite-past memory traces in this intriguing proposal. A lower bound from Fano's inequality shows that, on highly non-Markovian processes generated by large probabilistic state machines, next-generation reservoir computers with reasonably long memory traces have an error probability that is at least ~ 60% higher than the minimal attainable error probability in predicting the next observation. More generally, it appears that popular recurrent neural networks fall far short of optimally predicting such complex processes. These results highlight the need for a new generation of optimized recurrent neural network architectures. Alongside this finding, we present concentration-of-measure results for randomly-generated but complex processes. One conclusion is that large probabilistic state machines -- specifically, large $\epsilon$-machines -- are key to generating challenging and structurally-unbiased stimuli for ground-truthing recurrent neural network architectures.

翻译：循环神经网络被用于金融、气候、语言及众多其他领域的时间序列预测。储层计算是一种特别易于训练的循环神经网络形式。近期提出的"下一代"储层计算方案中，其记忆痕迹仅涉及有限数量的先前符号。我们探究了这种引人注目的方案中有限过去记忆痕迹的内在局限性。基于Fano不等式的下界表明，在大型概率状态机产生的高度非马尔可夫过程中，具有合理长记忆痕迹的下一代储层计算在预测下一个观测值时，其误差概率至少比最小可达误差概率高出约60%。更普遍而言，流行的循环神经网络在最优预测此类复杂过程方面仍存在显著不足。这些结果凸显了开发新一代优化循环神经网络架构的必要性。伴随这一发现，我们提出了针对随机生成但复杂过程的测度集中结果。结论之一表明：大型概率状态机——特别是大型$\epsilon$-机——是生成具有挑战性且结构无偏刺激以验证循环神经网络架构的关键工具。