Control design for robotic systems is complex and often requires solving an optimization to follow a trajectory accurately. Online optimization approaches like Model Predictive Control (MPC) have been shown to achieve great tracking performance, but require high computing power. Conversely, learning-based offline optimization approaches, such as Reinforcement Learning (RL), allow fast and efficient execution on the robot but hardly match the accuracy of MPC in trajectory tracking tasks. In systems with limited compute, such as aerial vehicles, an accurate controller that is efficient at execution time is imperative. We propose an Analytic Policy Gradient (APG) method to tackle this problem. APG exploits the availability of differentiable simulators by training a controller offline with gradient descent on the tracking error. We address training instabilities that frequently occur with APG through curriculum learning and experiment on a widely used controls benchmark, the CartPole, and two common aerial robots, a quadrotor and a fixed-wing drone. Our proposed method outperforms both model-based and model-free RL methods in terms of tracking error. Concurrently, it achieves similar performance to MPC while requiring more than an order of magnitude less computation time. Our work provides insights into the potential of APG as a promising control method for robotics. To facilitate the exploration of APG, we open-source our code and make it available at https://github.com/lis-epfl/apg_trajectory_tracking.

翻译：机器人系统的控制设计复杂且通常需要求解优化问题以实现精确轨迹跟踪。基于在线优化的方法，如模型预测控制（MPC），已被证明具有出色的跟踪性能，但需消耗大量计算资源。相反，基于学习的离线优化方法（如强化学习）虽能在机器人上实现快速高效执行，但在轨迹跟踪任务中难以匹敌MPC的精度。在计算资源受限的系统（如飞行器）中，兼顾执行效率与精度的控制器至关重要。本文提出一种解析策略梯度（APG）方法来解决该问题。APG利用可微分模拟器的可用性，通过梯度下降法离线训练控制器以最小化跟踪误差。我们通过课程学习应对APG中频繁出现的训练不稳定性，并在广泛使用的控制基准任务CartPole以及两种常见空中机器人（四旋翼飞行器和固定翼无人机）上进行实验。实验结果表明，所提方法在跟踪误差上优于基于模型和无模型的强化学习方法；同时，其性能与MPC相当，但计算时间降低一个数量级以上。本研究揭示了APG作为机器人领域有前景的控制方法的潜力。为促进APG的探索，我们开源了代码，并公开于https://github.com/lis-epfl/apg_trajectory_tracking。