This paper presents a novel algorithm for the continuous control of dynamical systems that combines Trajectory Optimization (TO) and Reinforcement Learning (RL) in a single framework. The motivations behind this algorithm are the two main limitations of TO and RL when applied to continuous nonlinear systems to minimize a non-convex cost function. Specifically, TO can get stuck in poor local minima when the search is not initialized close to a "good" minimum. On the other hand, when dealing with continuous state and control spaces, the RL training process may be excessively long and strongly dependent on the exploration strategy. Thus, our algorithm learns a "good" control policy via TO-guided RL policy search that, when used as initial guess provider for TO, makes the trajectory optimization process less prone to converge to poor local optima. Our method is validated on several reaching problems featuring non-convex obstacle avoidance with different dynamical systems, including a car model with 6D state, and a 3-joint planar manipulator. Our results show the great capabilities of CACTO in escaping local minima, while being more computationally efficient than the Deep Deterministic Policy Gradient (DDPG) and Proximal Policy Optimization (PPO) RL algorithms.

翻译：本文提出了一种新颖的算法，用于连续动力学系统的控制，该算法将轨迹优化（TO）与强化学习（RL）统一在同一框架中。该算法的动机源于TO和RL在应用于连续非线性系统以最小化非凸代价函数时的两个主要局限性。具体而言，当搜索未从接近“良好”最小值处初始化时，TO可能陷入较差的局部极小值。另一方面，在处理连续状态和控制空间时，RL训练过程可能过于漫长，且高度依赖探索策略。因此，我们的算法通过TO引导的RL策略搜索学习一个“良好”的控制策略，该策略作为TO的初始猜测提供者时，可使轨迹优化过程更不易收敛至较差的局部最优解。该方法在多个包含非凸避障的到达问题中进行了验证，涉及不同的动力学系统，包括具有六维状态的汽车模型和三关节平面机械臂。结果表明，CACTO在逃离局部最小值方面表现出强大能力，同时其计算效率优于深度确定性策略梯度（DDPG）和近端策略优化（PPO）强化学习算法。