While deep reinforcement learning (RL) agents outperform humans on an increasing number of tasks, training them requires data equivalent to decades of human gameplay. Recent hierarchical RL methods have increased sample efficiency by incorporating information inherent to the structure of the decision problem but at the cost of having to discover or use human-annotated sub-goals that guide the learning process. We show that intentions of human players, i.e. the precursor of goal-oriented decisions, can be robustly predicted from eye gaze even for the long-horizon sparse rewards task of Montezuma's Revenge - one of the most challenging RL tasks in the Atari2600 game suite. We propose Int-HRL: Hierarchical RL with intention-based sub-goals that are inferred from human eye gaze. Our novel sub-goal extraction pipeline is fully automatic and replaces the need for manual sub-goal annotation by human experts. Our evaluations show that replacing hand-crafted sub-goals with automatically extracted intentions leads to a HRL agent that is significantly more sample efficient than previous methods.

翻译：尽管深度强化学习（RL）智能体在越来越多的任务中超越人类表现，其训练所需的数据量相当于人类数十年的游戏经验。近期分层强化学习方法通过融入决策问题结构固有的信息提升了样本效率，但代价是需要发现或使用人工标注的子目标来引导学习过程。我们证明，人类玩家的意图（即目标导向决策的前兆）能够通过眼动追踪被稳健预测，即使是在《蒙提祖玛的复仇》这一长期稀疏奖励任务中——这是Atari2600游戏套件中最具挑战性的强化学习任务之一。我们提出Int-HRL：一种基于人类眼动推断的意图子目标的分层强化学习方法。我们提出的新型子目标提取流程完全自动化，取代了人类专家手动标注子目标的需求。实验评估表明，用自动提取的意图替代手工设计的子目标，可使分层强化学习智能体比先前方法具有显著更高的样本效率。