Problems which require both long-horizon planning and continuous control capabilities pose significant challenges to existing reinforcement learning agents. In this paper we introduce a novel hierarchical reinforcement learning agent which links temporally extended skills for continuous control with a forward model in a symbolic discrete abstraction of the environment's state for planning. We term our agent SEADS for Symbolic Effect-Aware Diverse Skills. We formulate an objective and corresponding algorithm which leads to unsupervised learning of a diverse set of skills through intrinsic motivation given a known state abstraction. The skills are jointly learned with the symbolic forward model which captures the effect of skill execution in the state abstraction. After training, we can leverage the skills as symbolic actions using the forward model for long-horizon planning and subsequently execute the plan using the learned continuous-action control skills. The proposed algorithm learns skills and forward models that can be used to solve complex tasks which require both continuous control and long-horizon planning capabilities with high success rate. It compares favorably with other flat and hierarchical reinforcement learning baseline agents and is successfully demonstrated with a real robot.

翻译：同时需要长时域规划与连续控制能力的问题对现有强化学习智能体构成了重大挑战。本文提出了一种新型分层强化学习智能体，它将用于连续控制的时域扩展技能与环境状态符号离散抽象中的前向模型相结合，以实现规划。我们将该智能体命名为SEADS（即符号效应感知多样化技能）。我们提出了一个目标函数及其对应算法，通过已知状态抽象下的内在动机实现多样化技能的无监督学习。这些技能与符号前向模型联合学习，该模型捕获了技能执行在状态抽象中的效应。训练完成后，我们可将这些技能作为符号动作，利用前向模型进行长时域规划，随后使用所学的连续动作控制技能执行该规划。所提出的算法能够学习可用于解决复杂任务的技能和前向模型，这些任务需要同时具备连续控制与长时域规划能力，且成功率较高。该方法与其他平面及分层强化学习基线智能体相比表现更优，并已在实际机器人上成功验证。