Recurrent Neural Networks (RNN) are ubiquitous computing systems for sequences and multivariate time series data. While several robust architectures of RNN are known, it is unclear how to relate RNN initialization, architecture, and other hyperparameters with accuracy for a given task. In this work, we propose to treat RNN as dynamical systems and to correlate hyperparameters with accuracy through Lyapunov spectral analysis, a methodology specifically designed for nonlinear dynamical systems. To address the fact that RNN features go beyond the existing Lyapunov spectral analysis, we propose to infer relevant features from the Lyapunov spectrum with an Autoencoder and an embedding of its latent representation (AeLLE). Our studies of various RNN architectures show that AeLLE successfully correlates RNN Lyapunov spectrum with accuracy. Furthermore, the latent representation learned by AeLLE is generalizable to novel inputs from the same task and is formed early in the process of RNN training. The latter property allows for the prediction of the accuracy to which RNN would converge when training is complete. We conclude that representation of RNN through Lyapunov spectrum along with AeLLE provides a novel method for organization and interpretation of variants of RNN architectures.

翻译：循环神经网络（RNN）是处理序列与多元时间序列数据的普适计算系统。尽管已存在多种稳健的RNN架构，但如何将RNN初始化、架构及超参数与特定任务的准确率相关联尚不清晰。本文提出将RNN视为动力系统，通过专为非线性动力系统设计的李雅普诺夫谱分析方法，实现超参数与准确率的相关性建模。针对RNN特征超越现有李雅普诺夫谱分析范畴的问题，我们提出利用自编码器及其潜在表征嵌入（AeLLE）从李雅普诺夫谱中推断相关特征。对多种RNN架构的研究表明，AeLLE能成功地将RNN李雅普诺夫谱与准确率相关联。此外，AeLLE习得的潜在表征可泛化至同一任务的新输入，并在RNN训练早期阶段形成。后者特性使得我们能够在训练完成前预测RNN将收敛到的准确率。我们得出结论：通过李雅普诺夫谱结合AeLLE表征RNN，为组织与解读RNN架构变体提供了新方法。