Deep reinforcement learning (RL) has been endowed with high expectations in tackling challenging manipulation tasks in an autonomous and self-directed fashion. Despite the significant strides made in the development of reinforcement learning, the practical deployment of this paradigm is hindered by at least two barriers, namely, the engineering of a reward function and ensuring the safety guaranty of learning-based controllers. In this paper, we address these challenging limitations by proposing a framework that merges a reinforcement learning \lstinline[columns=fixed]{planner} that is trained using sparse rewards with a model predictive controller (MPC) \lstinline[columns=fixed]{actor}, thereby offering a safe policy. On the one hand, the RL \lstinline[columns=fixed]{planner} learns from sparse rewards by selecting intermediate goals that are easy to achieve in the short term and promising to lead to target goals in the long term. On the other hand, the MPC \lstinline[columns=fixed]{actor} takes the suggested intermediate goals from the RL \lstinline[columns=fixed]{planner} as the input and predicts how the robot's action will enable it to reach that goal while avoiding any obstacles over a short period of time. We evaluated our method on four challenging manipulation tasks with dynamic obstacles and the results demonstrate that, by leveraging the complementary strengths of these two components, the agent can solve manipulation tasks in complex, dynamic environments safely with a $100\%$ success rate. Videos are available at \url{https://videoviewsite.wixsite.com/mpc-hgg}.

翻译：深度强化学习（RL）在自主处理高难度操作任务方面展现出巨大潜力。尽管强化学习技术取得了显著进展，其实际部署仍面临至少两大障碍：奖励函数的设计工程问题，以及确保基于学习控制器的安全性保障。本文针对这些挑战性约束提出一种融合框架，通过将采用稀疏奖励训练的强化学习规划器（planner）与模型预测控制（MPC）执行器（actor）相结合，实现安全策略。一方面，强化学习规划器通过选择短期易实现且长期有望导向目标的中间目标点，从稀疏奖励中进行学习；另一方面，模型预测控制执行器将强化学习规划器建议的中间目标作为输入，预测机器人动作如何在短周期内规避障碍物并达成目标。我们在四个含有动态障碍物的高难度操作任务上评估该方法，结果表明：通过利用两类组件的互补优势，智能体能够在复杂动态环境中以100%的成功率安全完成操作任务。视频演示见\url{https://videoviewsite.wixsite.com/mpc-hgg}。