Safe reinforcement learning (Safe RL) aims to maximize expected return while satisfying safety constraints, typically modeled as Constrained Markov Decision Processes (CMDPs). While primal-dual methods scale well to deep RL, they often suffer from delayed constraint correction, leading to oscillatory behavior and prolonged safety violations. In this paper, we propose Constraint-Sensitive Policy Optimization (CSPO), a first-order primal-dual method that incorporates local constraint sensitivity into policy updates. CSPO augments the primal objective with a constraint-sensitive correction derived from the shortest signed distance to the safety boundary, enabling smarter recovery steps back to safety, compensating for delayed Lagrange multiplier updates, reducing oscillations near the boundary, and preserving the KKT solutions of the original constrained problem. Experiments on navigation and locomotion benchmarks demonstrate that CSPO achieves faster safety recovery and high reward preservation, resulting in higher constrained returns compared to state-of-the-art primal-dual and penalty-based methods

翻译：安全强化学习旨在满足安全约束的同时最大化期望回报，通常建模为约束马尔可夫决策过程。虽然原始-对偶方法可扩展到深度强化学习，但常因约束修正延迟导致振荡行为与持续的安全违规。本文提出约束敏感策略优化（CSPO），这是一种结合局部约束敏感性的原始-对偶一阶方法。CSPO通过安全边界的最短有符号距离推导出约束敏感修正项来扩充原始目标，从而实现更智能的安全恢复步骤、补偿延迟的拉格朗日乘子更新、减少边界附近的振荡，并保留原约束问题的KKT解。导航与运动基准实验表明，与最先进的原始-对偶方法和基于惩罚的方法相比，CSPO实现了更快的安全恢复与高回报保持，从而获得更高的约束回报。