In this paper we propose real-time recurrent reinforcement learning (RTRRL), a biologically plausible approach to solving discrete and continuous control tasks in partially-observable markov decision processes (POMDPs). RTRRL consists of three parts: (1) a Meta-RL RNN architecture, implementing on its own an actor-critic algorithm; (2) an outer reinforcement learning algorithm, exploiting temporal difference learning and dutch eligibility traces to train the Meta-RL network; and (3) random-feedback local-online (RFLO) learning, an online automatic differentiation algorithm for computing the gradients with respect to parameters of the network.Our experimental results show that by replacing the optimization algorithm in RTRRL with the biologically implausible back propagation through time (BPTT), or real-time recurrent learning (RTRL), one does not improve returns, while matching the computational complexity for BPTT, and even increasing complexity for RTRL. RTRRL thus serves as a model of learning in biological neural networks, mimicking reward pathways in the basal ganglia.

翻译：本文提出实时循环强化学习（RTRRL），一种在部分可观测马尔可夫决策过程（POMDPs）中解决离散与连续控制任务的生物合理性方法。RTRRL包含三个组成部分：（1）元强化学习循环神经网络架构，其自身可实现演员-评论家算法；（2）外层强化学习算法，利用时序差分学习与荷兰资格迹来训练元强化学习网络；（3）随机反馈局部在线（RFLO）学习，一种用于计算网络参数梯度的在线自动微分算法。实验结果表明，将RTRRL中的优化算法替换为生物不可信的随时间反向传播（BPTT）或实时循环学习（RTRL）后，收益未见提升，且计算复杂度与BPTT持平，甚至因使用RTRL而增加。因此，RTRRL可作为生物神经网络学习模型，模拟基底神经节中的奖赏通路。