The deployment of robots in uncontrolled environments requires them to operate robustly under previously unseen scenarios, like irregular terrain and wind conditions. Unfortunately, while rigorous safety frameworks from robust optimal control theory scale poorly to high-dimensional nonlinear dynamics, control policies computed by more tractable "deep" methods lack guarantees and tend to exhibit little robustness to uncertain operating conditions. This work introduces a novel approach enabling scalable synthesis of robust safety-preserving controllers for robotic systems with general nonlinear dynamics subject to bounded modeling error by combining game-theoretic safety analysis with adversarial reinforcement learning in simulation. Following a soft actor-critic scheme, a safety-seeking fallback policy is co-trained with an adversarial "disturbance" agent that aims to invoke the worst-case realization of model error and training-to-deployment discrepancy allowed by the designer's uncertainty. While the learned control policy does not intrinsically guarantee safety, it is used to construct a real-time safety filter (or shield) with robust safety guarantees based on forward reachability rollouts. This shield can be used in conjunction with a safety-agnostic control policy, precluding any task-driven actions that could result in loss of safety. We evaluate our learning-based safety approach in a 5D race car simulator, compare the learned safety policy to the numerically obtained optimal solution, and empirically validate the robust safety guarantee of our proposed safety shield against worst-case model discrepancy.

翻译：在非受控环境中部署机器人要求其在此前未曾见过的场景（如不规则地形和风力条件）下也能稳健运行。然而，鲁棒最优控制理论中的严谨安全框架难以扩展到高维非线性动力学系统，而由更易处理的"深度"方法计算出的控制策略缺乏可证明的保证，且对不确定运行条件的鲁棒性往往较差。本文通过将博弈论安全分析与模拟环境中的对抗性强化学习相结合，提出了一种新颖方法，能够为受有界建模误差影响的通用非线性动力学机器人系统，实现可扩展的鲁棒安全控制器综合。遵循软演员-评论家框架，一个追求安全的备选策略与一个对抗性"干扰"智能体进行协同训练——该干扰智能体旨在引发设计者不确定性所允许的最坏情况模型误差及训练-部署偏差。尽管学习得到的控制策略本身并不具备内在安全性保证，但可基于前向可达性推演，构建一个具有鲁棒安全保证的实时安全滤波器（或称"护盾"）。该护盾可与无关安全的控制策略协同使用，阻止任何可能引发安全损失的任务驱动行为。我们在五维赛车模拟器中评估了这种基于学习的安全方法，将学习到的安全策略与数值最优解进行了对比，并通过实验验证了所提安全护盾在应对最坏情况模型偏差时的鲁棒安全保证。