This is a tutorial paper on Recurrent Neural Network (RNN), Long Short-Term Memory Network (LSTM), and their variants. We start with a dynamical system and backpropagation through time for RNN. Then, we discuss the problems of gradient vanishing and explosion in long-term dependencies. We explain close-to-identity weight matrix, long delays, leaky units, and echo state networks for solving this problem. Then, we introduce LSTM gates and cells, history and variants of LSTM, and Gated Recurrent Units (GRU). Finally, we introduce bidirectional RNN, bidirectional LSTM, and the Embeddings from Language Model (ELMo) network, for processing a sequence in both directions.

翻译：本文是一篇关于循环神经网络（RNN）、长短期记忆网络（LSTM）及其变体的教程性论文。我们首先从动力系统和反向传播通过时间（BPTT）的角度介绍RNN。接着，讨论长期依赖中梯度消失与梯度爆炸的问题，并阐释接近单位权重矩阵、长延迟、泄漏单元以及回声状态网络等解决方案。然后，介绍LSTM的门控机制与细胞结构、LSTM的发展历史及其变体，以及门控循环单元（GRU）。最后，介绍双向RNN、双向LSTM以及用于双向序列处理的ELMo（Embeddings from Language Model）网络。