In this paper, we revisit the recurrent back-propagation (RBP) algorithm, discuss the conditions under which it applies as well as how to satisfy them in deep neural networks. We show that RBP can be unstable and propose two variants based on conjugate gradient on the normal equations (CG-RBP) and Neumann series (Neumann-RBP). We further investigate the relationship between Neumann-RBP and back propagation through time (BPTT) and its truncated version (TBPTT). Our Neumann-RBP has the same time complexity as TBPTT but only requires constant memory, whereas TBPTT's memory cost scales linearly with the number of truncation steps. We examine all RBP variants along with BPTT and TBPTT in three different application domains: associative memory with continuous Hopfield networks, document classification in citation networks using graph neural networks and hyperparameter optimization for fully connected networks. All experiments demonstrate that RBPs, especially the Neumann-RBP variant, are efficient and effective for optimizing convergent recurrent neural networks. Code is released at: \url{https://github.com/lrjconan/RBP}.

翻译：在本文中,我们重新审视了反复出现的回调(RBP)算法(RBP),讨论其适用条件以及如何在深神经网络中满足这些算法。我们显示RBP可能不稳定,并根据正常方程(CG-RBP)和Neumann序列(Neumann-RBP)的共振梯度提出两种变异。我们进一步调查Neumann-RBP和通过时间反向传播(BBPTT)及其脱轨版本(TBBBTT)之间的关系。我们的Neumann-RBPT与T一样复杂,但仅需要恒定的内存,而TBPTT的内存成本尺度与转轨步骤的数目线性。我们审查了三个不同的应用领域的所有RBPTF变量:与连续的Hopfield网络的联系记忆、使用图表神经网络的文件分类和完全连接网络的超光度仪优化。所有实验都表明,特别是Neurmann-RBPBPF的变式,对于优化集中的经常性神经网络是高效和有效的。代码。代码公布在以下网址上:\urlallasmus/gus/gus/conjurbus/conjrb/coml。