Safety is critical to broadening the application of reinforcement learning (RL). Often, we train RL agents in a controlled environment, such as a laboratory, before deploying them in the real world. However, the real-world target task might be unknown prior to deployment. Reward-free RL trains an agent without the reward to adapt quickly once the reward is revealed. We consider the constrained reward-free setting, where an agent (the guide) learns to explore safely without the reward signal. This agent is trained in a controlled environment, which allows unsafe interactions and still provides the safety signal. After the target task is revealed, safety violations are not allowed anymore. Thus, the guide is leveraged to compose a safe behaviour policy. Drawing from transfer learning, we also regularize a target policy (the student) towards the guide while the student is unreliable and gradually eliminate the influence of the guide as training progresses. The empirical analysis shows that this method can achieve safe transfer learning and helps the student solve the target task faster.

翻译：安全性对于拓展强化学习的应用至关重要。通常，我们在受控环境（如实验室）中训练强化学习智能体，然后将其部署到现实世界中。然而，现实世界的目标任务可能在部署前仍是未知的。无奖励强化学习训练智能体时不借助奖励信号，以便在奖励函数揭示后能够快速适应。我们考虑约束条件下的无奖励设置，其中智能体（引导者）学习在无奖励信号的情况下安全探索。该智能体在受控环境中训练，允许不安全交互并仍能提供安全信号。当目标任务揭示后，安全违规将不再被允许。因此，我们利用引导者来构建一个安全行为策略。借鉴迁移学习的思想，我们在目标策略（学生模型）尚不可靠时，将其向引导者正则化，并随着训练进程逐步消除引导者的影响。实证分析表明，该方法能够实现安全迁移学习，并帮助学生模型更快地解决目标任务。