Reinforcement learning (RL) has made significant strides in various complex domains. However, identifying an effective policy via RL often necessitates extensive exploration. Imitation learning aims to mitigate this issue by using expert demonstrations to guide exploration. In real-world scenarios, one often has access to multiple suboptimal black-box experts, rather than a single optimal oracle. These experts do not universally outperform each other across all states, presenting a challenge in actively deciding which oracle to use and in which state. We introduce MAPS and MAPS-SE, a class of policy improvement algorithms that perform imitation learning from multiple suboptimal oracles. In particular, MAPS actively selects which of the oracles to imitate and improve their value function estimates, and MAPS-SE additionally leverages an active state exploration criterion to determine which states one should explore. We provide a comprehensive theoretical analysis and demonstrate that MAPS and MAPS-SE enjoy sample efficiency advantage over the state-of-the-art policy improvement algorithms. Empirical results show that MAPS-SE significantly accelerates policy optimization via state-wise imitation learning from multiple oracles across a broad spectrum of control tasks in the DeepMind Control Suite. Our code is publicly available at: https://github.com/ripl/maps.

翻译：强化学习（RL）已在多种复杂领域取得显著进展，但通过RL确定有效策略通常需要大量探索。模仿学习旨在通过利用专家演示指导探索来缓解这一问题。在实际场景中，我们往往能获取多个次优的黑盒专家而非单一最优专家。这些专家并非在所有状态下均优于彼此，这为如何主动决策在何种状态下使用哪个专家带来了挑战。本文提出MAPS和MAPS-SE这两类策略改进算法，它们能够从多个次优专家策略中进行模仿学习。具体而言，MAPS主动选择需要模仿的专家并改进其价值函数估计，而MAPS-SE则额外利用主动状态探索准则来确定应探索的状态。我们提供了完整的理论分析，证明MAPS和MAPS-SE在样本效率上优于现有最优策略改进算法。实验结果表明，在DeepMind控制套件涵盖的广泛控制任务中，MAPS-SE通过从多个专家进行逐状态模仿学习显著加速了策略优化。我们的代码已开源：https://github.com/ripl/maps。