In reservoir computing, an input sequence is processed by a recurrent neural network, the reservoir, which transforms it into a spatial pattern that a shallow readout network can then exploit for tasks such as memorization and time-series prediction or classification. Echo state networks (ESN) are a model class in which the reservoir is a traditional artificial neural network. This class contains many model types, each with sets of hyperparameters. Selecting models and parameter settings for particular applications requires a theory for predicting and comparing performances. Here, we demonstrate that recent developments of perceptron theory can be used to predict the memory capacity and accuracy of a wide variety of ESN models, including reservoirs with linear neurons, sigmoid nonlinear neurons, different types of recurrent matrices, and different types of readout networks. Across thirty variants of ESNs, we show that empirical results consistently confirm the theory's predictions. As a practical demonstration, the theory is used to optimize memory capacity of an ESN in the entire joint parameter space. Further, guided by the theory, we propose a novel ESN model with a readout network that does not require training, and which outperforms earlier ESN models without training. Finally, we characterize the geometry of the readout networks in ESNs, which reveals that many ESN models exhibit a similar regular simplex geometry as has been observed in the output weights of deep neural networks.

翻译：在储层计算中，输入序列通过一个循环神经网络（即储层）进行处理，该网络将输入转换为空间模式，随后可由浅层读出网络利用该模式完成记忆、时间序列预测或分类等任务。回声状态网络（ESN）是一类模型，其储层采用传统人工神经网络结构。此类模型包含多种类型，每种类型均具有多组超参数。为特定应用选择模型及参数设置需要一套能够预测和比较性能的理论框架。本文证明，感知器理论的最新进展可用于预测多种ESN模型的记忆容量与精度，包括采用线性神经元、S型非线性神经元、不同类型循环矩阵及不同读出网络的储层结构。通过对三十种ESN变体的实验，我们证明经验结果始终与理论预测相符。作为实际应用演示，该理论被用于在完整联合参数空间中优化ESN的记忆容量。进一步地，在理论指导下，我们提出了一种新型ESN模型，其读出网络无需训练即可超越早期未经训练的ESN模型。最后，我们刻画了ESN中读出网络的几何特性，发现许多ESN模型呈现出与深度神经网络输出权重中观察到的类似正则单纯形几何结构。