Online learning holds the promise of enabling efficient long-term credit assignment in recurrent neural networks. However, current algorithms fall short of offline backpropagation by either not being scalable or failing to learn long-range dependencies. Here we present a high-performance online learning algorithm that merely doubles the memory and computational requirements of a single inference pass. We achieve this by leveraging independent recurrent modules in multi-layer networks, an architectural motif that has recently been shown to be particularly powerful. Experiments on synthetic memory problems and on the challenging long-range arena benchmark suite reveal that our algorithm performs competitively, establishing a new standard for what can be achieved through online learning. This ability to learn long-range dependencies offers a new perspective on learning in the brain and opens a promising avenue in neuromorphic computing.

翻译：在线学习有望实现循环神经网络中高效的长时信用分配。然而，当前算法在可扩展性不足或无法学习长程依赖关系方面，仍落后于离线反向传播。本文提出一种高性能在线学习算法，其内存和计算需求仅相当于单次推理的两倍。我们通过利用多层网络中独立的循环模块（这一近期被证明尤为强大的架构模式）实现了这一目标。在合成记忆问题和具有挑战性的长程竞技场基准测试套件上的实验表明，我们的算法表现具有竞争力，为在线学习所能达到的水平建立了新标准。这种学习长程依赖关系的能力为大脑中的学习机制提供了新视角，并为神经形态计算开辟了有前景的发展方向。