An open research question in robotics is how to combine the benefits of model-free reinforcement learning (RL) - known for its strong task performance and flexibility in optimizing general reward formulations - with the robustness and online replanning capabilities of model predictive control (MPC). This paper provides an answer by introducing a new framework called Actor-Critic Model Predictive Control. The key idea is to embed a differentiable MPC within an actor-critic RL framework. The proposed approach leverages the short-term predictive optimization capabilities of MPC with the exploratory and end-to-end training properties of RL. The resulting policy effectively manages both short-term decisions through the MPC-based actor and long-term prediction via the critic network, unifying the benefits of both model-based control and end-to-end learning. We validate our method in both simulation and the real world with a quadcopter platform across various high-level tasks. We show that the proposed architecture can achieve real-time control performance, learn complex behaviors via trial and error, and retain the robustness inherent to MPC.

翻译：机器人学中的一个开放研究问题是如何融合无模型强化学习（以其强大的任务性能和优化一般奖励公式的灵活性而闻名）与模型预测控制（具有鲁棒性和在线重规划能力）的优势。本文通过引入一种称为“演员-评论家模型预测控制”的新框架来回答这一问题。其核心思想是将可微分模型预测控制嵌入到演员-评论家强化学习框架中。所提出的方法利用了模型预测控制的短期预测优化能力以及强化学习的探索性与端到端训练特性。由此产生的策略通过基于模型预测控制的演员有效管理短期决策，并通过评论家网络进行长期预测，从而统一了基于模型的控制与端到端学习的优势。我们在仿真和真实世界的四旋翼飞行器平台上，针对多种高级任务验证了该方法。结果表明，所提出的架构能够实现实时控制性能，通过试错学习复杂行为，并保留模型预测控制固有的鲁棒性。