Guaranteeing safe behaviour of reinforcement learning (RL) policies poses significant challenges for safety-critical applications, despite RL's generality and scalability. To address this, we propose a new approach to apply verification methods from control theory to learned value functions. By analyzing task structures for safety preservation, we formalize original theorems that establish links between value functions and control barrier functions. Further, we propose novel metrics for verifying value functions in safe control tasks and practical implementation details to improve learning. Our work presents a novel method for certificate learning, which unlocks a diversity of verification techniques from control theory for RL policies, and marks a significant step towards a formal framework for the general, scalable, and verifiable design of RL-based control systems.

翻译：强化学习策略在确保安全行为方面面临重大挑战，尤其是在安全关键应用中，尽管其具有通用性和可扩展性。为此，我们提出了一种新方法，将控制理论中的验证方法应用于学习得到的价值函数。通过分析任务结构以保持安全性，我们形式化了原创性定理，建立了价值函数与控制障碍函数之间的联系。此外，我们提出了用于在安全控制任务中验证价值函数的新颖度量标准，以及改进学习的实际实现细节。我们的工作提出了一种新的证书学习方法，解锁了控制理论中用于强化学习策略的各种验证技术，并标志着向基于强化学习的控制系统实现通用、可扩展和可验证设计的正式框架迈出了重要一步。