We study the uniform approximation of echo state networks with randomly generated internal weights. These models, in which only the readout weights are optimized during training, have made empirical success in learning dynamical systems. Recent results showed that echo state networks with ReLU activation are universal. In this paper, we give an alternative construction and prove that the universality holds for general activation functions. Specifically, our main result shows that, under certain condition on the activation function, there exists a sampling procedure for the internal weights so that the echo state network can approximate any continuous casual time-invariant operators with high probability. In particular, for ReLU activation, we give explicit construction for these sampling procedures. We also quantify the approximation error of the constructed ReLU echo state networks for sufficiently regular operators.

翻译：本文研究了随机生成内部权重的回声状态网络的一致逼近性质。这类模型仅在训练过程中优化读出权重，已在学习动力系统方面取得实证成功。最新研究表明，采用ReLU激活函数的回声状态网络具有普适性。本文给出了一种替代构造方法，并证明该普适性可推广至一般激活函数。具体而言，我们的主要结论表明：在激活函数满足特定条件时，存在内部权重的抽样策略，使得回声状态网络能以高概率逼近任意连续因果时不变算子。特别地，针对ReLU激活函数，我们给出了这些抽样策略的显式构造，并量化了所构造的ReLU回声状态网络对充分正则算子的逼近误差。