Model-based next state prediction and state value prediction are slow to converge. To address these challenges, we do the following: i) Instead of a neural network, we do model-based planning using a parallel memory retrieval system (which we term the slow mechanism); ii) Instead of learning state values, we guide the agent's actions using goal-directed exploration, by using a neural network to choose the next action given the current state and the goal state (which we term the fast mechanism). The goal-directed exploration is trained online using hippocampal replay of visited states and future imagined states every single time step, leading to fast and efficient training. Empirical studies show that our proposed method has a 92% solve rate across 100 episodes in a dynamically changing grid world, significantly outperforming state-of-the-art actor critic mechanisms such as PPO (54%), TRPO (50%) and A2C (24%). Ablation studies demonstrate that both mechanisms are crucial. We posit that the future of Reinforcement Learning (RL) will be to model goals and sub-goals for various tasks, and plan it out in a goal-directed memory-based approach.

翻译：基于模型的下一状态预测与状态价值预测收敛缓慢。为解决这些挑战，我们采取以下措施：i) 采用并行记忆检索系统（称为慢速机制）替代神经网络进行基于模型的规划；ii) 通过目标导向探索引导智能体行为，即利用神经网络根据当前状态与目标状态选择下一步动作（称为快速机制），而非学习状态价值。该目标导向探索通过在每一个时间步利用海马体回放已访问状态与未来想象状态进行在线训练，从而实现快速高效的训练。实证研究表明，我们的方法在动态变化的网格世界中100回合求解率达92%，显著优于PPO（54%）、TRPO（50%）和A2C（24%）等先进演员-评论家机制。消融实验表明，两种机制均至关重要。我们认为，强化学习（RL）的未来发展方向是为不同任务建模目标与子目标，并通过基于目标导向记忆的方法进行规划。