Safety is critical when applying reinforcement learning (RL) to real-world problems. As a result, safe RL has emerged as a fundamental and powerful paradigm for optimizing an agent's policy while incorporating notions of safety. A prevalent safe RL approach is based on a constrained criterion, which seeks to maximize the expected cumulative reward subject to specific safety constraints. Despite recent effort to enhance safety in RL, a systematic understanding of the field remains difficult. This challenge stems from the diversity of constraint representations and little exploration of their interrelations. To bridge this knowledge gap, we present a comprehensive review of representative constraint formulations, along with a curated selection of algorithms designed specifically for each formulation. In addition, we elucidate the theoretical underpinnings that reveal the mathematical mutual relations among common problem formulations. We conclude with a discussion of the current state and future directions of safe reinforcement learning research.

翻译：安全性是将强化学习应用于现实问题时的关键因素。因此，安全强化学习已成为一类基础且重要的范式，旨在优化智能体策略的同时引入安全概念。一种主流的安全强化学习方法基于约束准则，即在满足特定安全约束的条件下最大化期望累积奖励。尽管近年来在增强强化学习安全性方面取得了一定进展，但对该领域的系统性理解仍存在困难。这一挑战源于约束表示方式的多样性以及对其相互关系的探索不足。为弥合这一知识空白，我们系统综述了具有代表性的约束形式，并专门为每种形式精选了对应的算法。此外，我们阐明了理论框架，揭示了常见问题表述之间数学层面的相互关联。最后，我们探讨了安全强化学习研究的现状与未来方向。