Recurrent Neural Networks (RNNs) are widely used for online regression due to their ability to generalize nonlinear temporal dependencies. As an RNN model, Long-Short-Term-Memory Networks (LSTMs) are commonly preferred in practice, as these networks are capable of learning long-term dependencies while avoiding the vanishing gradient problem. However, due to their large number of parameters, training LSTMs requires considerably longer training time compared to simple RNNs (SRNNs). In this paper, we achieve the online regression performance of LSTMs with SRNNs efficiently. To this end, we introduce a first-order training algorithm with a linear time complexity in the number of parameters. We show that when SRNNs are trained with our algorithm, they provide very similar regression performance with the LSTMs in two to three times shorter training time. We provide strong theoretical analysis to support our experimental results by providing regret bounds on the convergence rate of our algorithm. Through an extensive set of experiments, we verify our theoretical work and demonstrate significant performance improvements of our algorithm with respect to LSTMs and the other state-of-the-art learning models.

翻译：经常性神经网络(RNN)由于能够概括非线性时间依赖性,因此被广泛用于在线回归。作为一个RNN模式,长期短期-中期网络(LSTM)在实际中普遍被青睐,因为这些网络能够学习长期依赖性,同时避免消失的梯度问题。然而,由于其参数众多,培训LSTMS需要比简单的RNNS(SRNS)长得多的培训时间。在本文中,我们高效率地实现了LSTMS与SRNS的在线回归性表现。为此,我们引入了在参数数量上具有线性时间复杂性的第一阶培训算法。我们表明,当SRNNS接受我们的算法培训时,它们可以在两到三倍的较短的培训时间里提供与LSTMs非常相似的回归性表现。我们提供了强有力的理论分析,通过在我们的算法趋同率上提供遗憾的界限来支持我们的实验结果。我们通过广泛的一系列实验,核查我们的理论工作,并展示我们对LSTMs和其他州学习模型进行的重大绩效改进。