While reinforcement learning (RL) has shown promising performance, its sample complexity continues to be a substantial hurdle, restricting its broader application across a variety of domains. Imitation learning (IL) utilizes oracles to improve sample efficiency, yet it is often constrained by the quality of the oracles deployed. which actively interleaves between IL and RL based on an online estimate of their performance. RPI draws on the strengths of IL, using oracle queries to facilitate exploration, an aspect that is notably challenging in sparse-reward RL, particularly during the early stages of learning. As learning unfolds, RPI gradually transitions to RL, effectively treating the learned policy as an improved oracle. This algorithm is capable of learning from and improving upon a diverse set of black-box oracles. Integral to RPI are Robust Active Policy Selection (RAPS) and Robust Policy Gradient (RPG), both of which reason over whether to perform state-wise imitation from the oracles or learn from its own value function when the learner's performance surpasses that of the oracles in a specific state. Empirical evaluations and theoretical analysis validate that RPI excels in comparison to existing state-of-the-art methodologies, demonstrating superior performance across various benchmark domains.

翻译：尽管强化学习（RL）已展现出令人瞩目的性能，但其样本复杂度仍是制约其更广泛领域应用的关键障碍。模仿学习（IL）通过利用专家系统提升样本效率，然而其性能往往受限于所采用专家系统的质量。本文提出鲁棒策略改进（RPI）算法——一种基于对IL与RL性能进行在线评估，主动在两者间进行动态切换的方法。RPI充分发挥IL优势，利用专家查询促进探索，这在稀疏奖励强化学习（尤其在早期学习阶段）中极具挑战性。随着学习进程推进，RPI逐步过渡至RL，将已学习的策略视为改进后的专家系统。该算法能通过学习并超越多种黑箱专家系统实现性能提升。其核心组件——鲁棒主动策略选择（RAPS）与鲁棒策略梯度（RPG）——均基于特定状态下的学习器性能是否超越专家系统，对执行状态级专家模仿或基于自身价值函数学习进行推理。实证评估与理论分析表明，RPI在多个基准测试领域均展现出优于现有最先进方法的卓越性能。