Despite the promising results achieved, state-of-the-art interactive reinforcement learning schemes rely on passively receiving supervision signals from advisor experts, in the form of either continuous monitoring or pre-defined rules, which inevitably result in a cumbersome and expensive learning process. In this paper, we introduce a novel initiative advisor-in-the-loop actor-critic framework, termed as Ask-AC, that replaces the unilateral advisor-guidance mechanism with a bidirectional learner-initiative one, and thereby enables a customized and efficacious message exchange between learner and advisor. At the heart of Ask-AC are two complementary components, namely action requester and adaptive state selector, that can be readily incorporated into various discrete actor-critic architectures. The former component allows the agent to initiatively seek advisor intervention in the presence of uncertain states, while the latter identifies the unstable states potentially missed by the former especially when environment changes, and then learns to promote the ask action on such states. Experimental results on both stationary and non-stationary environments and across different actor-critic backbones demonstrate that the proposed framework significantly improves the learning efficiency of the agent, and achieves the performances on par with those obtained by continuous advisor monitoring.

翻译：尽管取得了令人鼓舞的成果，但最先进的交互式强化学习方案依赖于被动地从顾问专家处接收监督信号，形式为持续监控或预定义规则，这不可避免地导致了繁琐且昂贵的学习过程。本文提出了一种新颖的主动顾问参与的角色-评论家框架，命名为Ask-AC，该框架用双向的学习者主动机制取代了单方面的顾问引导机制，从而实现学习者和顾问之间的定制化且有效的消息交换。Ask-AC的核心是两个互补的组件，即动作请求器和自适应状态选择器，它们可以轻松集成到各种离散角色-评论家架构中。前者允许智能体在遇到不确定状态时主动寻求顾问干预，而后者识别前者可能遗漏的不稳定状态——尤其在环境变化时——并学习在此类状态上促进请求动作的触发。在静态和非静态环境中，基于不同角色-评论家主干的实验结果表明，所提出的框架显著提高了智能体的学习效率，并达到了与持续顾问监控相当的性能。