An open research question in robotics is how to combine the benefits of model-free reinforcement learning (RL) - known for its strong task performance and flexibility in optimizing general reward formulations - with the robustness and online replanning capabilities of model predictive control (MPC). This paper provides an answer by introducing a new framework called Actor-Critic Model Predictive Control. The key idea is to embed a differentiable MPC within an actor-critic RL framework. The proposed approach leverages the short-term predictive optimization capabilities of MPC with the exploratory and end-to-end training properties of RL. The resulting policy effectively manages both short-term decisions through the MPC-based actor and long-term prediction via the critic network, unifying the benefits of both model-based control and end-to-end learning. We validate our method in both simulation and the real world with a quadcopter platform across various high-level tasks. We show that the proposed architecture can achieve real-time control performance, learn complex behaviors via trial and error, and retain the robustness inherent to MPC.

翻译：机器人学中的一个开放研究问题是如何结合无模型强化学习（以强任务性能和优化通用奖励公式的灵活性著称）与模型预测控制的鲁棒性和在线重规划能力。本文通过引入一种名为"主动-评论模型预测控制"的新框架给出了答案。其核心思想是在主动-评论强化学习框架中嵌入一个可微分的模型预测控制。该方法利用模型预测控制的短期预测优化能力与强化学习的探索性和端到端训练特性。由此产生的策略通过基于模型预测控制的主动部分有效管理短期决策，并通过评论网络进行长期预测，统一了基于模型的控制和端到端学习的优势。我们在仿真和真实世界中以四旋翼飞行平台为载体，在多种高级任务上验证了该方法。结果表明，所提出的架构能够实现实时控制性能，通过试错学习复杂行为，并保留模型预测控制固有的鲁棒性。