Recent advances in reinforcement learning, for partially-observable Markov decision processes (POMDPs), rely on the biologically implausible backpropagation through time algorithm (BPTT) to perform gradient-descent optimisation. In this paper we propose a novel reinforcement learning algorithm that makes use of random feedback local online learning (RFLO), a biologically plausible approximation of realtime recurrent learning (RTRL) to compute the gradients of the parameters of a recurrent neural network in an online manner. By combining it with TD($\lambda$), a variant of temporaldifference reinforcement learning with eligibility traces, we create a biologically plausible, recurrent actor-critic algorithm, capable of solving discrete and continuous control tasks in POMDPs. We compare BPTT, RTRL and RFLO as well as different network architectures, and find that RFLO can perform just as well as RTRL while exceeding even BPTT in terms of complexity. The proposed method, called real-time recurrent reinforcement learning (RTRRL), serves as a model of learning in biological neural networks mimicking reward pathways in the mammalian brain.

翻译：近期，针对部分可观测马尔可夫决策过程（POMDPs）的强化学习进展，依赖于生物学上不可信的随时间反向传播算法（BPTT）进行梯度下降优化。本文提出了一种新颖的强化学习算法，该算法利用随机反馈局部在线学习（RFLO）——一种生物学上可信的实时递归学习（RTRL）近似方法——以在线方式计算递归神经网络参数的梯度。通过将其与TD($\lambda$)（一种结合资格迹的时间差分强化学习变体）相结合，我们构建了一个生物学上可信的递归演员-评论家算法，能够解决POMDPs中的离散和连续控制任务。我们比较了BPTT、RTRL、RFLO以及不同网络架构，发现RFLO在性能上与RTRL相当，且在计算复杂度上甚至优于BPTT。所提出的方法称为实时递归强化学习（RTRRL），可作为生物神经网络中学习过程的模型，模拟哺乳动物大脑中的奖赏通路。