One major limitation to the applicability of Reinforcement Learning (RL) to many practical domains is the large number of samples required to learn an optimal policy. To address this problem and improve learning efficiency, we consider a linear hierarchy of abstraction layers of the Markov Decision Process (MDP) underlying the target domain. Each layer is an MDP representing a coarser model of the one immediately below in the hierarchy. In this work, we propose a novel form of Reward Shaping where the solution obtained at the abstract level is used to offer rewards to the more concrete MDP, in such a way that the abstract solution guides the learning in the more complex domain. In contrast with other works in Hierarchical RL, our technique has few requirements in the design of the abstract models and it is also tolerant to modeling errors, thus making the proposed approach practical. We formally analyze the relationship between the abstract models and the exploration heuristic induced in the lower-level domain. Moreover, we prove that the method guarantees optimal convergence and we demonstrate its effectiveness experimentally.

翻译：强化学习（RL）在众多实际领域应用的一大限制，在于学习最优策略所需的大量样本。为解决此问题并提高学习效率，我们考虑目标领域底层马尔可夫决策过程（MDP）的抽象层线性层次结构。每一层都是一个MDP，表示层次结构中紧邻下层模型的粗化版本。在本工作中，我们提出一种新型奖励塑形方法，利用抽象层获得的解为更具体的MDP提供奖励，从而引导抽象解在更复杂领域中的学习过程。与分层强化学习领域的其他工作不同，我们的技术在抽象模型设计方面需求极少，且对建模误差具有较高容忍度，这使得所提方法具备实用性。我们形式化分析了抽象模型与其在下层领域诱导的探索启发式之间的关联。此外，我们证明了该方法保证最优收敛性，并通过实验验证了其有效性。