We introduce recurrent additive networks (RANs), a new gated RNN which is distinguished by the use of purely additive latent state updates. At every time step, the new state is computed as a gated component-wise sum of the input and the previous state, without any of the non-linearities commonly used in RNN transition dynamics. We formally show that RAN states are weighted sums of the input vectors, and that the gates only contribute to computing the weights of these sums. Despite this relatively simple functional form, experiments demonstrate that RANs perform on par with LSTMs on benchmark language modeling problems. This result shows that many of the non-linear computations in LSTMs and related networks are not essential, at least for the problems we consider, and suggests that the gates are doing more of the computational work than previously understood.

翻译：我们引入了经常性添加剂网络(RANs),这是一个新的封闭式RNN(RANs),其特点是使用纯添加剂潜伏状态更新。在每一个步骤中,新状态都是作为输入和先前状态的封闭部分总和来计算,没有在RNN过渡动态中常用的任何非线性数据。我们正式表明,RAN国家是输入矢量的加权总和,而大门只有助于计算这些总量的重量。尽管这种功能形式相对简单,但实验表明RANs在基准语言建模问题上的表现与LSTMs相同。这一结果表明,LSTMS和相关网络中的许多非线性计算并不必要,至少对于我们所考虑的问题来说是如此,并且表明大门的计算工作比以前所理解的要多。