Exploration algorithms for reinforcement learning typically replace or augment the reward function with an additional ``intrinsic'' reward that trains the agent to seek previously unseen states of the environment. Here, we consider an exploration algorithm that exploits meta-learning, or learning to learn, such that the agent learns to maximize its exploration progress within a single episode, even between epochs of training. The agent learns a policy that aims to minimize the probability density of new observations with respect to all of its memories. In addition, it receives as feedback evaluations of the current observation density and retains that feedback in a recurrent network. By remembering trajectories of density, the agent learns to navigate a complex and growing landscape of familiarity in real-time, allowing it to maximize its exploration progress even in completely novel states of the environment for which its policy has not been trained.

翻译：强化学习中的探索算法通常通过引入额外的"内在"奖励来替代或增强原有的奖励函数，从而训练智能体寻找环境中先前未访问过的状态。本文提出一种利用元学习（即学会学习）的探索算法，使智能体能够在单个训练周期内（甚至在训练轮次之间）持续学习如何最大化其探索进度。该智能体通过学习一种策略，旨在最小化新观测值相对于其所有记忆的概率密度。此外，系统会接收当前观测密度的评估反馈，并将该反馈保存在循环神经网络中。通过记忆密度轨迹，智能体能够实时学习在复杂且持续扩展的熟悉度景观中导航，从而即使在环境完全新颖、其策略未经训练的状态下，也能最大化探索进度。