Despite the considerable potential of reinforcement learning (RL), robotic control tasks predominantly rely on imitation learning (IL) due to its better sample efficiency. However, it is costly to collect comprehensive expert demonstrations that enable IL to generalize to all possible scenarios, and any distribution shift would require recollecting data for finetuning. Therefore, RL is appealing if it can build upon IL as an efficient autonomous self-improvement procedure. We propose imitation bootstrapped reinforcement learning (IBRL), a novel framework for sample-efficient RL with demonstrations that first trains an IL policy on the provided demonstrations and then uses it to propose alternative actions for both online exploration and bootstrapping target values. Compared to prior works that oversample the demonstrations or regularize RL with an additional imitation loss, IBRL is able to utilize high quality actions from IL policies since the beginning of training, which greatly accelerates exploration and training efficiency. We evaluate IBRL on 6 simulation and 3 real-world tasks spanning various difficulty levels. IBRL significantly outperforms prior methods and the improvement is particularly more prominent in harder tasks.

翻译：尽管强化学习（RL）具有巨大潜力，机器人控制任务仍主要依赖模仿学习（IL），因其样本效率更高。然而，收集能让IL泛化到所有可能场景的全面专家演示成本高昂，且任何分布偏移都需要重新收集数据进行微调。因此，若RL能基于IL构建高效的自主自我改进流程，则具有吸引力。我们提出模仿引导的强化学习（IBRL），这是一种基于演示的样本高效RL新框架，它首先在提供的演示上训练IL策略，然后利用该策略为在线探索和目标值引导提出替代动作。与先前过度采样演示或通过额外模仿损失正则化RL的方法相比，IBRL能在训练初期就利用IL策略的高质量动作，从而极大加速探索和训练效率。我们在6个仿真任务和3个难度各异真实世界任务上评估IBRL。IBRL显著优于先前方法，且在更困难任务中提升尤为突出。