This paper proposes an exploration-efficient Deep Reinforcement Learning with Reference policy (DRLR) framework for learning robotics tasks that incorporates demonstrations. The DRLR framework is developed based on an algorithm called Imitation Bootstrapped Reinforcement Learning (IBRL). We propose to improve IBRL by modifying the action selection module. The proposed action selection module provides a calibrated Q-value, which mitigates the bootstrapping error that otherwise leads to inefficient exploration. Furthermore, to prevent the RL policy from converging to a sub-optimal policy, SAC is used as the RL policy instead of TD3. The effectiveness of our method in mitigating bootstrapping error and preventing overfitting is empirically validated by learning two robotics tasks: bucket loading and open drawer, which require extensive interactions with the environment. Simulation results also demonstrate the robustness of the DRLR framework across tasks with both low and high state-action dimensions, and varying demonstration qualities. To evaluate the developed framework on a real-world industrial robotics task, the bucket loading task is deployed on a real wheel loader. The sim2real results validate the successful deployment of the DRLR framework.

翻译：本文提出了一种融合演示的探索高效深度强化学习参考策略框架，用于学习机器人任务。该框架基于一种称为模仿引导强化学习的算法构建。我们建议通过修改动作选择模块来改进该算法。所提出的动作选择模块提供经过校准的Q值，从而减轻了原本会导致低效探索的引导误差。此外，为防止强化学习策略收敛至次优策略，我们采用SAC而非TD3作为强化学习策略。通过让机器人学习需要与环境进行大量交互的铲斗装载和抽屉开启两项任务，我们实证验证了本方法在减轻引导误差和防止过拟合方面的有效性。仿真结果还表明，该框架在状态-动作维度高低不同、演示质量各异的任务中均表现出鲁棒性。为在实际工业机器人任务中评估所开发框架，我们将铲斗装载任务部署于真实轮式装载机。仿真到现实的迁移结果验证了该框架的成功部署。