A popular framework for enforcing safe actions in Reinforcement Learning (RL) is Constrained RL, where trajectory based constraints on expected cost (or other cost measures) are employed to enforce safety and more importantly these constraints are enforced while maximizing expected reward. Most recent approaches for solving Constrained RL convert the trajectory based cost constraint into a surrogate problem that can be solved using minor modifications to RL methods. A key drawback with such approaches is an over or underestimation of the cost constraint at each state. Therefore, we provide an approach that does not modify the trajectory based cost constraint and instead imitates ``good'' trajectories and avoids ``bad'' trajectories generated from incrementally improving policies. We employ an oracle that utilizes a reward threshold (which is varied with learning) and the overall cost constraint to label trajectories as ``good'' or ``bad''. A key advantage of our approach is that we are able to work from any starting policy or set of trajectories and improve on it. In an exhaustive set of experiments, we demonstrate that our approach is able to outperform top benchmark approaches for solving Constrained RL problems, with respect to expected cost, CVaR cost, or even unknown cost constraints.

翻译：在强化学习（RL）中执行安全动作的流行框架是约束强化学习，它采用基于轨迹的期望成本（或其他成本度量）约束来确保安全性，更重要的是，在最大化期望奖励的同时强制执行这些约束。大多数求解约束强化学习的最新方法将基于轨迹的成本约束转化为一个替代问题，只需对强化学习方法进行微调即可求解。这类方法的一个关键缺陷是，每个状态下成本约束存在高估或低估问题。因此，我们提出一种不修改基于轨迹成本约束的方法，而是通过模仿渐进改进策略生成的“好”轨迹并规避“坏”轨迹。我们采用一个利用奖励阈值（随学习过程动态调整）和整体成本约束的预言机来标记轨迹为“好”或“坏”。我们方法的一个关键优势在于，能够从任何初始策略或轨迹集出发并加以改进。在详尽的实验中，我们证明该方法在期望成本、条件风险价值（CVaR）成本甚至未知成本约束方面，均能超越求解约束强化学习问题的顶尖基准方法。