This paper presents a novel algorithm for the continuous control of dynamical systems that combines Trajectory Optimization (TO) and Reinforcement Learning (RL) in a single framework. The motivations behind this algorithm are the two main limitations of TO and RL when applied to continuous nonlinear systems to minimize a non-convex cost function. Specifically, TO can get stuck in poor local minima when the search is not initialized close to a "good" minimum. On the other hand, when dealing with continuous state and control spaces, the RL training process may be excessively long and strongly dependent on the exploration strategy. Thus, our algorithm learns a "good" control policy via TO-guided RL policy search that, when used as initial guess provider for TO, makes the trajectory optimization process less prone to converge to poor local optima. Our method is validated on several reaching problems featuring non-convex obstacle avoidance with different dynamical systems, including a car model with 6D state, and a 3-joint planar manipulator. Our results show the great capabilities of CACTO in escaping local minima, while being more computationally efficient than the Deep Deterministic Policy Gradient (DDPG) and Proximal Policy Optimization (PPO) RL algorithms.

翻译：本文提出了一种新颖的算法，用于动态系统的连续控制，该算法将轨迹优化（TO）与强化学习（RL）整合在单一框架中。该算法的核心动机源于TO和RL在应用于连续非线性系统以最小化非凸成本函数时的两大局限性。具体而言，当搜索初始点未接近“理想”最小值时，TO可能陷入糟糕的局部最小值。另一方面，在处理连续状态和动作空间时，RL训练过程可能异常漫长且严重依赖于探索策略。因此，我们的算法通过TO引导的RL策略搜索学习一个“理想”控制策略，该策略作为TO的初始猜测提供者时，能使轨迹优化过程不易陷入较差的局部最优。该方法在多个涉及非凸障碍物避让的到达任务中进行了验证，这些任务涵盖不同动力学系统，包括六维状态的车辆模型和三关节平面机械臂。结果表明，CACTO在逃离局部最小值方面展现出卓越能力，同时计算效率优于深度确定性策略梯度（DDPG）和近端策略优化（PPO）等RL算法。