Interactive reinforcement learning has shown promise in learning complex robotic tasks. However, the process can be human-intensive due to the requirement of large amount of interactive feedback. This paper presents a new method that uses scores provided by humans, instead of pairwise preferences, to improve the feedback efficiency of interactive reinforcement learning. Our key insight is that scores can yield significantly more data than pairwise preferences. Specifically, we require a teacher to interactively score the full trajectories of an agent to train a behavioral policy in a sparse reward environment. To avoid unstable scores given by human negatively impact the training process, we propose an adaptive learning scheme. This enables the learning paradigm to be insensitive to imperfect or unreliable scores. We extensively evaluate our method on robotic locomotion and manipulation tasks. The results show that the proposed method can efficiently learn near-optimal policies by adaptive learning from scores, while requiring less feedback compared to pairwise preference learning methods. The source codes are publicly available at https://github.com/SSKKai/Interactive-Scoring-IRL.

翻译：交互式强化学习在学习复杂机器人任务方面展现出潜力。然而，由于需要大量交互式反馈，该过程可能对人力和时间消耗较大。本文提出一种新方法，利用人类提供的分数（而非成对偏好）来提高交互式强化学习的反馈效率。我们的关键洞察是：分数可以比成对偏好产生显著更多的数据。具体而言，我们要求教师对智能体的完整轨迹进行交互式评分，以在稀疏奖励环境中训练行为策略。为避免人类给出的不稳定分数对训练过程产生负面影响，我们提出一种自适应学习方案，使学习范式对不完美或不可靠的分数具有鲁棒性。我们在机器人运动控制和操作任务上进行了广泛评估。结果表明，与成对偏好学习方法相比，所提方法通过自适应分数学习，能以更少的反馈高效学习接近最优的策略。源代码已公开于https://github.com/SSKKai/Interactive-Scoring-IRL。