Continuous-time nonlinear optimal control problems hold great promise in real-world applications. After decades of development, reinforcement learning (RL) has achieved some of the greatest successes as a general nonlinear control design method. However, a recent comprehensive analysis of state-of-the-art continuous-time RL (CT-RL) methods, namely, adaptive dynamic programming (ADP)-based CT-RL algorithms, reveals they face significant design challenges due to their complexity, numerical conditioning, and dimensional scaling issues. Despite advanced theoretical results, existing ADP CT-RL synthesis methods are inadequate in solving even small, academic problems. The goal of this work is thus to introduce a suite of new CT-RL algorithms for control of affine nonlinear systems. Our design approach relies on two important factors. First, our methods are applicable to physical systems that can be partitioned into smaller subproblems. This constructive consideration results in reduced dimensionality and greatly improved intuitiveness of design. Second, we introduce a new excitation framework to improve persistence of excitation (PE) and numerical conditioning performance via classical input/output insights. Such a design-centric approach is the first of its kind in the ADP CT-RL community. In this paper, we progressively introduce a suite of (decentralized) excitable integral reinforcement learning (EIRL) algorithms. We provide convergence and closed-loop stability guarantees, and we demonstrate these guarantees on a significant application problem of controlling an unstable, nonminimum phase hypersonic vehicle (HSV).

翻译：连续时间非线性最优控制问题在实际应用中具有广阔前景。经过数十年发展，强化学习作为通用非线性控制设计方法取得了重大成功。然而，最新对先进连续时间强化学习方法——即基于自适应动态规划的连续时间强化学习算法——的全面分析表明，这类方法因复杂性、数值条件性和维度扩展问题面临显著设计挑战。尽管具备先进的理论成果，现有自适应动态规划类连续时间强化学习综合方法甚至难以解决小规模学术问题。因此，本文旨在引入一套面向仿射非线性系统控制的新型连续时间强化学习算法。我们的设计方法依赖于两个关键因素：首先，该方法适用于可分解为若干子问题的物理系统，这种结构化设计可降低维度并显著提升设计直观性；其次，我们提出新型激励框架，通过经典输入/输出视角改进持续激励与数值条件性表现。这种以设计为中心的方法在自适应动态规划类连续时间强化学习领域尚属首创。本文逐步引入一套（分散式）可激励积分强化学习算法，提供收敛性与闭环稳定性保证，并通过控制不稳定非最小相位高超声速飞行器的重大应用问题验证了这些保证。