Reservoir computing is a machine learning framework where the readouts from a nonlinear system (the reservoir) are trained so that the output from the reservoir, when forced with an input signal, reproduces a desired output signal. A common implementation of reservoir computers is to use a recurrent neural network as the reservoir. The design of this network can have significant effects on the performance of the reservoir computer. In this paper we study the effect of the node activation function on the ability of reservoir computers to learn and predict chaotic time series. We find that the Forecast Horizon (FH), the time during which the reservoir's predictions remain accurate, can vary by an order of magnitude across a set of 16 activation functions used in machine learning. By using different functions from this set, and by modifying their parameters, we explore whether the entropy of node activation levels or the curvature of the activation functions determine the predictive ability of the reservoirs. We find that the FH is low when the activation function is used in a region where it has low curvature, and a positive correlation between curvature and FH. For the activation functions studied we find that the largest FH generally occurs at intermediate levels of the entropy of node activation levels. Our results show that the performance of reservoir computers is very sensitive to the activation function shape. Therefore, modifying this shape in hyperparameter optimization algorithms can lead to improvements in reservoir computer performance.

翻译：储备池计算是一种机器学习框架，通过训练非线性系统（储备池）的读出层，使得当输入信号驱动储备池时，其输出能够复现期望的输出信号。储备池计算机的常见实现方式是将循环神经网络用作储备池。该网络的设计对储备池计算机的性能具有显著影响。本文研究了节点激活函数对储备池计算机学习和预测混沌时间序列能力的影响。我们发现，在机器学习中常用的16种激活函数中，预测时域（即储备池预测保持准确的时间）可相差一个数量级。通过采用这些函数的不同变体并调整其参数，我们探究了节点激活水平的熵或激活函数的曲率是否决定了储备池的预测能力。研究发现，当激活函数在低曲率区域使用时，预测时域较低，且曲率与预测时域呈正相关。在所研究的激活函数中，最大预测时域通常出现在节点激活水平熵的中间区间。我们的结果表明，储备池计算机的性能对激活函数的形状极为敏感。因此，在超参数优化算法中调整这一形状可有效提升储备池计算机的性能。