Recurrent networks are typically trained with backpropagation through time (BPTT). However, BPTT requires storing the history of all states in the network and then replaying them sequentially backwards in time. This computation appears extremely implausible for the brain to implement. Real Time Recurrent Learning (RTRL) proposes an mathematically equivalent alternative where gradient information is propagated forwards in time locally alongside the regular forward pass, however it has significantly greater computational complexity than BPTT which renders it impractical for large networks. E-prop proposes an approximation of RTRL which reduces its complexity to the level of BPTT while maintaining a purely online forward update which can be implemented by an eligibility trace at each synapse. However, works on RTRL and E-prop ubiquitously investigate learning in a single layer with recurrent dynamics. However, learning in the brain spans multiple layers and consists of both hierarchal dynamics in depth as well as time. In this mathematical note, we extend the E-prop framework to handle arbitrarily deep networks, deriving a novel recursion relationship across depth which extends the eligibility traces of E-prop to deeper layers. Our results thus demonstrate an online learning algorithm can perform accurate credit assignment across both time and depth simultaneously, allowing the training of deep recurrent networks without backpropagation through time.

翻译：循环网络通常通过时间反向传播（BPTT）进行训练。然而，BPTT需要存储网络中所有状态的历史记录，然后按时间顺序反向回放。这种计算方式在大脑中实现显得极不合理。实时循环学习（RTRL）提出了一种数学上等效的替代方案，其中梯度信息与常规前向传递同时沿时间向前局部传播，但其计算复杂度显著高于BPTT，使得该方法在大规模网络中不具实用性。E-prop提出了RTRL的一种近似方法，将其复杂度降低至BPTT水平，同时保持纯在线前向更新机制，该机制可通过每个突触的资格迹实现。然而，现有关于RTRL和E-prop的研究普遍局限于具有循环动态的单层网络学习。而大脑中的学习过程跨越多个层次，同时包含深度方向的层级动态和时间动态。在本数学笔记中，我们将E-prop框架扩展至任意深度网络，推导出跨深度的新型递归关系，从而将E-prop的资格迹延伸至更深层。我们的研究结果证明，在线学习算法能够同时跨时间和深度进行精确的信用分配，实现在无需时间反向传播的情况下训练深度循环网络。