In many practical control applications, the performance level of a closed-loop system degrades over time due to the change of plant characteristics. Thus, there is a strong need for redesigning a controller without going through the system modeling process, which is often difficult for closed-loop systems. Reinforcement learning (RL) is one of the promising approaches that enable model-free redesign of optimal controllers for nonlinear dynamical systems based only on the measurement of the closed-loop system. However, the learning process of RL usually requires a considerable number of trial-and-error experiments using the poorly controlled system that may accumulate wear on the plant. To overcome this limitation, we propose a model-free two-step design approach that improves the transient learning performance of RL in an optimal regulator redesign problem for unknown nonlinear systems. Specifically, we first design a linear control law that attains some degree of control performance in a model-free manner, and then, train the nonlinear optimal control law with online RL by using the designed linear control law in parallel. We introduce an offline RL algorithm for the design of the linear control law and theoretically guarantee its convergence to the LQR controller under mild assumptions. Numerical simulations show that the proposed approach improves the transient learning performance and efficiency in hyperparameter tuning of RL.

翻译：在许多实际控制应用中，由于被控对象特性随时间变化，闭环系统的性能水平会逐渐下降。因此，迫切需要一种无需经过系统建模过程（这对闭环系统通常较为困难）即可重设计控制器的方法。强化学习是一种有前景的方法，它仅基于闭环系统的测量数据，即可实现非线性动态系统最优控制器的无模型重设计。然而，强化学习的学习过程通常需要使用性能较差的受控系统进行大量试错实验，这可能导致被控对象磨损累积。为克服这一局限，我们提出一种无模型两步设计方法，旨在提升未知非线性系统最优调节器重设计问题中强化学习的瞬态学习性能。具体而言，我们首先以无模型方式设计一个能达到一定控制性能的线性控制律，然后通过在线强化学习，并行利用该线性控制律训练非线性最优控制律。我们引入一种离线强化学习算法用于线性控制律设计，并在温和假设下从理论上保证其收敛至LQR控制器。数值仿真表明，所提方法能够改善强化学习的瞬态学习性能与超参数调优效率。