This paper puts forward the concept that learning to take safe actions in unknown environments, even with probability one guarantees, can be achieved without the need for an unbounded number of exploratory trials. This is indeed possible, provided that one is willing to navigate trade-offs between optimality, level of exposure to unsafe events, and the maximum detection time of unsafe actions. We illustrate this concept in two complementary settings. We first focus on the canonical multi-armed bandit problem and study the intrinsic trade-offs of learning safety in the presence of uncertainty. Under mild assumptions on sufficient exploration, we provide an algorithm that provably detects all unsafe machines in an (expected) finite number of rounds. The analysis also unveils a trade-off between the number of rounds needed to secure the environment and the probability of discarding safe machines. We then consider the problem of finding optimal policies for a Markov Decision Process (MDP) with almost sure constraints. We show that the action-value function satisfies a barrier-based decomposition which allows for the identification of feasible policies independently of the reward process. Using this decomposition, we develop a Barrier-learning algorithm, that identifies such unsafe state-action pairs in a finite expected number of steps. Our analysis further highlights a trade-off between the time lag for the underlying MDP necessary to detect unsafe actions, and the level of exposure to unsafe events. Simulations corroborate our theoretical findings, further illustrating the aforementioned trade-offs, and suggesting that safety constraints can speed up the learning process.

翻译：本文提出一个概念：在未知环境中学习采取安全行动，即使要求概率为一的保证，也无需进行无限次探索试验。只要愿意在最优性、不安全事件暴露程度以及不安全行动最大检测时间之间权衡取舍，这确实是可能的。我们在两个互补的场景中阐述这一概念。首先，聚焦于经典的多臂老虎机问题，研究在不确定性下学习安全性的内在权衡。在对充分探索的温和假设下，我们提出了一种算法，该算法可证明在（期望）有限轮数内检测出所有不安全的机器。分析还揭示了确保环境安全所需的轮数与丢弃安全机器的概率之间的权衡。接着，我们考虑在几乎必然约束下为马尔可夫决策过程（MDP）寻找最优策略的问题。我们证明动作价值函数满足一种基于障碍的分解，该分解允许独立于奖励过程识别可行策略。利用这一分解，我们开发了一种障碍学习算法，该算法能在有限期望步数内识别出此类不安全的状态-动作对。我们的分析进一步强调了底层MDP检测不安全行动所需的时间延迟与暴露于不安全事件的程度之间的权衡。仿真结果验证了我们的理论发现，进一步说明了上述权衡，并表明安全约束能够加速学习过程。