In this work, we investigate the memory capability of recurrent neural networks (RNNs), where this capability is defined as a function that maps an element in a sequence to the current output. We first analyze the system function of a recurrent neural network (RNN) cell, and provide analytical results for three RNNs. They are the simple recurrent neural network (SRN), the long short-term memory (LSTM), and the gated recurrent unit (GRU). Based on the analysis, we propose a new design to extend the memory length of a cell, and call it the extended long short-term memory (ELSTM). Next, we present a dependent bidirectional recurrent neural network (DBRNN) for the sequence-in-sequence-out (SISO) problem, which is more robust to previous erroneous predictions. Extensive experiments are carried out on different language tasks to demonstrate the superiority of our proposed ELSTM and DBRNN solutions.

翻译：在这项工作中,我们调查经常性神经网络(RNNs)的记忆能力,这种能力被界定为根据当前输出的顺序绘制一个元素的函数。我们首先分析一个经常性神经网络(RNN)细胞的系统功能,并为三个RNS提供分析结果。它们是简单的经常性神经网络(SRN)、长期短期内存(LSTM)和大门内经常性单元(GRU)。根据分析,我们提出了延长一个细胞内存长度的新设计,并称之为延长的短期内存(ELSTM )。接下来,我们提出了一个附属的双向性双向经常性神经网络(DBRNN),用于处理顺序内退出(SISO)问题,这比以往错误的预测更为有力。在不同的语言任务上进行了广泛的实验,以展示我们提议的ELTM和DBRNN的解决方案的优越性。