It is quite challenging to ensure the safety of reinforcement learning (RL) agents in an unknown and stochastic environment under hard constraints that require the system state not to reach certain specified unsafe regions. Many popular safe RL methods such as those based on the Constrained Markov Decision Process (CMDP) paradigm formulate safety violations in a cost function and try to constrain the expectation of cumulative cost under a threshold. However, it is often difficult to effectively capture and enforce hard reachability-based safety constraints indirectly with such constraints on safety violation costs. In this work, we leverage the notion of barrier function to explicitly encode the hard safety constraints, and given that the environment is unknown, relax them to our design of \emph{generative-model-based soft barrier functions}. Based on such soft barriers, we propose a safe RL approach that can jointly learn the environment and optimize the control policy, while effectively avoiding unsafe regions with safety probability optimization. Experiments on a set of examples demonstrate that our approach can effectively enforce hard safety constraints and significantly outperform CMDP-based baseline methods in system safe rate measured via simulations.

翻译：在硬约束条件下，确保强化学习（RL）智能体在未知且随机环境中的安全性极具挑战性，这些硬约束要求系统状态不得进入特定不安全区域。许多流行的安全强化学习方法（如基于约束马尔可夫决策过程（CMDP）范式的方法）将安全违规纳入代价函数，并试图将累计代价的期望值限制在阈值以下。然而，通过此类安全违规代价的约束，往往难以有效捕捉和强制执行基于可达性的硬安全约束。在本工作中，我们利用屏障函数的概念显式编码硬安全约束，并在环境未知的情况下将其松弛为我们设计的"基于生成模型的软屏障函数"。基于此类软屏障，我们提出了一种安全强化学习方法，该方法可联合学习环境并优化控制策略，同时通过安全概率优化有效避开不安全区域。在示例集上的实验表明，我们的方法能够有效执行硬安全约束，并通过模拟评估的系统安全率显著优于基于CMDP的基线方法。