Shielding is a popular technique for achieving safe reinforcement learning (RL). However, classical shielding approaches come with quite restrictive assumptions making them difficult to deploy in complex environments, particularly those with continuous state or action spaces. In this paper we extend the more versatile approximate model-based shielding (AMBS) framework to the continuous setting. In particular we use Safety Gym as our test-bed, allowing for a more direct comparison of AMBS with popular constrained RL algorithms. We also provide strong probabilistic safety guarantees for the continuous setting. In addition, we propose two novel penalty techniques that directly modify the policy gradient, which empirically provide more stable convergence in our experiments.

翻译：屏蔽是实现安全强化学习（RL）的常用技术。然而，经典屏蔽方法具有相当严格的假设条件，使其难以部署在复杂环境中，尤其是连续状态或动作空间的环境。本文我们将更具通用性的近似模型屏蔽（AMBS）框架扩展至连续场景。具体而言，我们采用Safety Gym作为测试平台，以便更直接地将AMBS与流行的约束RL算法进行对比。同时，我们为连续场景提供了强概率安全保证。此外，我们提出两种新型惩罚技术，可直接修改策略梯度，实验证明这些技术能更稳定地实现收敛。