In this paper, we seek to learn a robot policy guaranteed to satisfy state constraints. To encourage constraint satisfaction, existing RL algorithms typically rely on Constrained Markov Decision Processes and discourage constraint violations through reward shaping. However, such soft constraints cannot offer verifiable safety guarantees. To address this gap, we propose POLICEd RL, a novel RL algorithm explicitly designed to enforce affine hard constraints in closed-loop with a black-box environment. Our key insight is to force the learned policy to be affine around the unsafe set and use this affine region as a repulsive buffer to prevent trajectories from violating the constraint. We prove that such policies exist and guarantee constraint satisfaction. Our proposed framework is applicable to both systems with continuous and discrete state and action spaces and is agnostic to the choice of the RL training algorithm. Our results demonstrate the capacity of POLICEd RL to enforce hard constraints in robotic tasks while significantly outperforming existing methods.

翻译：本文旨在学习一种保证满足状态约束的机器人策略。为鼓励约束满足，现有的强化学习算法通常依赖于约束马尔可夫决策过程，并通过奖励塑造来抑制约束违反。然而，此类软约束无法提供可验证的安全性保证。为解决这一不足，我们提出POLICEd RL——一种旨在与黑箱环境闭环中强制实施仿射硬约束的新型强化学习算法。我们的核心见解是迫使学习到的策略在非安全集附近保持仿射性，并利用该仿射区域作为排斥缓冲区，防止轨迹违反约束。我们证明了此类策略的存在性及其对约束满足的保证性。所提出的框架同时适用于连续和离散状态与动作空间的系统，且与具体强化学习训练算法的选择无关。实验结果表明，POLICEd RL能够在机器人任务中强制实施硬约束，同时显著优于现有方法。