Reinforcement learning (RL) involves performing exploratory actions in an unknown system. This can place a learning agent in dangerous and potentially catastrophic system states. Current approaches for tackling safe learning in RL simultaneously trade-off safe exploration and task fulfillment. In this paper, we introduce a new generation of RL solvers that learn to minimise safety violations while maximising the task reward to the extent that can be tolerated by the safe policy. Our approach introduces a novel two-player framework for safe RL called Distributive Exploration Safety Training Algorithm (DESTA). The core of DESTA is a game between two adaptive agents: Safety Agent that is delegated the task of minimising safety violations and Task Agent whose goal is to maximise the environment reward. Specifically, Safety Agent can selectively take control of the system at any given point to prevent safety violations while Task Agent is free to execute its policy at any other states. This framework enables Safety Agent to learn to take actions at certain states that minimise future safety violations, both during training and testing time, while Task Agent performs actions that maximise the task performance everywhere else. Theoretically, we prove that DESTA converges to stable points enabling safety violations of pretrained policies to be minimised. Empirically, we show DESTA's ability to augment the safety of existing policies and secondly, construct safe RL policies when the Task Agent and Safety Agent are trained concurrently. We demonstrate DESTA's superior performance against leading RL methods in Lunar Lander and Frozen Lake from OpenAI gym.

翻译：强化学习涉及在未知系统中执行探索性动作，这可能将学习智能体置于危险甚至灾难性的系统状态。当前应对安全强化学习的方法通常需要在安全探索与任务完成之间进行权衡。本文提出新一代强化学习求解器，能够在安全策略可容忍的范围内，在最大化任务奖励的同时最小化安全违规行为。我们的方法引入一种新颖的双智能体安全强化学习框架——分布式探索安全训练算法（DESTA）。DESTA的核心是两个自适应智能体之间的博弈：被授予最小化安全违规任务的安全智能体，以及以最大化环境奖励为目标的任务智能体。具体而言，安全智能体可在任意时刻选择性控制系统以防止安全违规，而任务智能体则在其他状态下自由执行其策略。该框架使安全智能体能够通过学习在特定状态采取行动，在训练和测试期间最小化未来安全违规，同时任务智能体在其他所有状态下执行最大化任务性能的动作。理论层面，我们证明DESTA能收敛至稳定点，使预训练策略的安全违规最小化。实验层面，我们展示了DESTA增强现有策略安全性的能力，并构建了任务智能体与安全智能体并行训练时的安全强化学习策略。在OpenAI gym平台的Lunar Lander和Frozen Lake环境中，DESTA的性能优于主流强化学习方法。