Despite the tremendous success of Reinforcement Learning (RL) algorithms in simulation environments, applying RL to real-world applications still faces many challenges. A major concern is safety, in another word, constraint satisfaction. State-wise constraints are one of the most common constraints in real-world applications and one of the most challenging constraints in Safe RL. Enforcing state-wise constraints is necessary and essential to many challenging tasks such as autonomous driving, robot manipulation. This paper provides a comprehensive review of existing approaches that address state-wise constraints in RL. Under the framework of State-wise Constrained Markov Decision Process (SCMDP), we will discuss the connections, differences, and trade-offs of existing approaches in terms of (i) safety guarantee and scalability, (ii) safety and reward performance, and (iii) safety after convergence and during training. We also summarize limitations of current methods and discuss potential future directions.

翻译：尽管强化学习算法在模拟环境中取得了巨大成功，但将强化学习应用于实际场景仍面临诸多挑战。核心问题之一是安全性，即约束满足。逐状态约束是实际应用中最常见的约束类型之一，同时也是安全强化学习中最具挑战性的约束之一。强制实施逐状态约束对于自动驾驶、机器人操作等众多复杂任务而言既必要又至关重要。本文全面综述了现有处理强化学习中逐状态约束的方法。在逐状态约束马尔可夫决策过程（SCMDP）框架下，我们将从以下三方面探讨现有方法的联系、差异与权衡：(i) 安全保证与可扩展性；(ii) 安全性与奖励性能；(iii) 收敛后及训练过程中的安全性。此外，我们总结了当前方法的局限性，并讨论了潜在的未来研究方向。