Recurrent neural networks are important tools for sequential data processing. However, they are notorious for problems regarding their training. Challenges include capturing complex relations between consecutive states and stability and efficiency of training. In this paper, we introduce a recurrent neural architecture called Deep Memory Update (DMU). It is based on updating the previous memory state with a deep transformation of the lagged state and the network input. The architecture is able to learn to transform its internal state using any nonlinear function. Its training is stable and fast due to relating its learning rate to the size of the module. Even though DMU is based on standard components, experimental results presented here confirm that it can compete with and often outperform state-of-the-art architectures such as Long Short-Term Memory, Gated Recurrent Units, and Recurrent Highway Networks.

翻译：循环神经网络是序列数据处理的重要工具，但其训练过程以困难著称，挑战包括捕获连续状态间的复杂关系以及训练稳定性与效率。本文提出一种名为“深度记忆更新”（Deep Memory Update, DMU）的循环神经架构。该架构通过将先前记忆状态与滞后状态及网络输入的深度变换相结合进行更新。它能够利用任意非线性函数学习转换内部状态。得益于学习率与模块规模相关联，其训练过程稳定且高效。尽管DMU基于标准组件，但实验结果表明，它能与长短期记忆网络（LSTM）、门控循环单元（GRU）及循环高速网络（Recurrent Highway Networks）等先进架构相竞争，且常能超越它们。