As autonomous robots move into complex, dynamic real-world environments, they must learn to navigate safely in real time, yet anticipating all possible behaviors is infeasible. We propose a composable, model-free reinforcement learning method that learns a value function and an optimal policy for each individual environment element (e.g., goal or obstacle) and composes them online to achieve goal reaching and collision avoidance. Assuming unknown nonlinear dynamics that evolve in continuous time and are input-affine, we derive a continuous-time Hamilton-Jacobi-Bellman (HJB) equation for the value function and show that the corresponding advantage function is quadratic in the action and optimal policy. Based on this structure, we introduce a model-free actor-critic algorithm that learns policies and value functions for static or moving obstacles using gradient descent. We then compose multiple reach/avoid models via a quadratically constrained quadratic program (QCQP), yielding formal obstacle-avoidance guarantees in terms of value-function level sets, providing a model-free alternative to CLF/CBF-based controllers. Simulations demonstrate improved performance over a PPO baseline applied to a discrete-time approximation.

翻译：随着自主机器人进入复杂、动态的现实世界环境，它们必须学会实时安全导航，然而预测所有可能行为是不可行的。我们提出了一种可组合的无模型强化学习方法，该方法为每个独立环境元素（例如目标或障碍物）学习一个价值函数和最优策略，并在线组合它们以实现目标到达和避碰。假设未知的非线性动力学在连续时间内演化且为仿射输入，我们推导了价值函数的连续时间 Hamilton-Jacobi-Bellman (HJB) 方程，并证明了相应的优势函数在动作和最优策略上是二次的。基于此结构，我们引入了一种无模型的演员-评论家算法，该算法使用梯度下降学习针对静态或移动障碍物的策略和价值函数。随后，我们通过一个二次约束二次规划 (QCQP) 组合多个到达/规避模型，从而在价值函数水平集方面提供形式化的避障保证，为基于 CLF/CBF 的控制器提供了一种无模型的替代方案。仿真结果表明，与应用于离散时间近似的 PPO 基线相比，本方法性能更优。