Online learning holds the promise of enabling efficient long-term credit assignment in recurrent neural networks. However, current algorithms fall short of offline backpropagation by either not being scalable or failing to learn long-range dependencies. Here we present a high-performance online learning algorithm that merely doubles the memory and computational requirements of a single inference pass. We achieve this by leveraging independent recurrent modules in multi-layer networks, an architectural motif that has recently been shown to be particularly powerful. Experiments on synthetic memory problems and on the challenging long-range arena benchmark suite reveal that our algorithm performs competitively, establishing a new standard for what can be achieved through online learning. This ability to learn long-range dependencies offers a new perspective on learning in the brain and opens a promising avenue in neuromorphic computing.

翻译：在线学习有望实现循环神经网络中长期信用分配的高效计算。然而，当前算法因缺乏可扩展性或无法学习长程依赖而落后于离线反向传播。本文提出一种高性能在线学习算法，其内存与计算需求仅为单次推理的两倍。我们通过利用多层网络中的独立循环模块实现这一目标——该架构特征近期被证明特别强大。在合成记忆问题与具有挑战性的长程竞技场基准套件上的实验表明，我们的算法表现优异，确立了在线学习所能达到的新标准。这种学习长程依赖的能力为大脑中的学习机制提供了新视角，并为神经形态计算开辟了有前景的方向。