This paper considers the problem of learning safe policies in the context of reinforcement learning (RL). In particular, we consider the notion of probabilistic safety. This is, we aim to design policies that maintain the state of the system in a safe set with high probability. This notion differs from cumulative constraints often considered in the literature. The challenge of working with probabilistic safety is the lack of expressions for their gradients. Indeed, policy optimization algorithms rely on gradients of the objective function and the constraints. To the best of our knowledge, this work is the first one providing such explicit gradient expressions for probabilistic constraints. It is worth noting that the gradient of this family of constraints can be applied to various policy-based algorithms. We demonstrate empirically that it is possible to handle probabilistic constraints in a continuous navigation problem.

翻译：本文研究了在强化学习（RL）背景下学习安全策略的问题。特别地，我们考虑了概率安全性的概念，即致力于设计能够以高概率将系统状态维持在安全集内的策略。这一概念不同于文献中常考虑的累积约束。处理概率安全性的挑战在于缺乏其梯度的解析表达式。事实上，策略优化算法依赖于目标函数和约束的梯度。据我们所知，本文是首个为概率约束提供此类显式梯度表达式的研究工作。值得注意的是，这类约束的梯度可应用于多种基于策略的算法。我们通过实验证明，在连续导航问题中处理概率约束是可行的。