Reinforcement learning (RL) has achieved promising results on most robotic control tasks. Safety of learning-based controllers is an essential notion of ensuring the effectiveness of the controllers. Current methods adopt whole consistency constraints during the training, thus resulting in inefficient exploration in the early stage. In this paper, we propose a Constrained Policy Optimization with Extra Safety Budget (ESB-CPO) algorithm to strike a balance between the exploration and the constraints. In the early stage, our method loosens the practical constraints of unsafe transitions (adding extra safety budget) with the aid of a new metric we propose. With the training process, the constraints in our optimization problem become tighter. Meanwhile, theoretical analysis and practical experiments demonstrate that our method gradually meets the cost limit's demand in the final training stage. When evaluated on Safety-Gym and Bullet-Safety-Gym benchmarks, our method has shown its advantages over baseline algorithms in terms of safety and optimality. Remarkably, our method gains remarkable performance improvement under the same cost limit compared with CPO algorithm.

翻译：强化学习在大多数机器人控制任务中已取得了有前景的成果。基于学习的控制器的安全性是确保控制器有效性的重要概念。现有方法在训练过程中采用全程一致性约束，从而导致早期阶段探索效率低下。本文提出一种带有额外安全预算的约束策略优化（ESB-CPO）算法，以平衡探索与约束之间的关系。在早期阶段，借助我们提出的新指标，该方法放宽了对不安全转移的实际约束（添加额外安全预算）。随着训练过程的进行，优化问题中的约束逐渐收紧。同时，理论分析与实践实验表明，我们的方法在最终训练阶段逐渐满足了成本限制的要求。在Safety-Gym和Bullet-Safety-Gym基准上的评估显示，该方法在安全性和最优性方面相较于基线算法具有优势。值得注意的是，在相同成本限制下，与CPO算法相比，我们的方法获得了显著的性能提升。