This paper puts forward the concept that learning to take safe actions in unknown environments, even with probability one guarantees, can be achieved without the need for an unbounded number of exploratory trials. This is indeed possible, provided that one is willing to navigate trade-offs between optimality, level of exposure to unsafe events, and the maximum detection time of unsafe actions. We illustrate this concept in two complementary settings. We first focus on the canonical multi-armed bandit problem and study the intrinsic trade-offs of learning safety in the presence of uncertainty. Under mild assumptions on sufficient exploration, we provide an algorithm that provably detects all unsafe machines in an (expected) finite number of rounds. The analysis also unveils a trade-off between the number of rounds needed to secure the environment and the probability of discarding safe machines. We then consider the problem of finding optimal policies for a Markov Decision Process (MDP) with almost sure constraints. We show that the action-value function satisfies a barrier-based decomposition which allows for the identification of feasible policies independently of the reward process. Using this decomposition, we develop a Barrier-learning algorithm, that identifies such unsafe state-action pairs in a finite expected number of steps. Our analysis further highlights a trade-off between the time lag for the underlying MDP necessary to detect unsafe actions, and the level of exposure to unsafe events. Simulations corroborate our theoretical findings, further illustrating the aforementioned trade-offs, and suggesting that safety constraints can speed up the learning process.

翻译：本文提出一个概念：在未知环境中学习安全行为，即使要求概率为1的保障，也无需进行无界次数的探索试验。这确实是可行的，前提是愿意在最优性、不安全事件的暴露程度以及不安全行为的最大检测时间之间进行权衡。我们在两个互补的场景中阐述了这一概念。首先，我们聚焦于经典的多臂赌博机问题，研究在不确定性下学习安全性的内在权衡。在充分探索的温和假设下，我们提出一种算法，能够在（期望）有限轮次内可证明地检测出所有不安全的机器。分析还揭示了确保环境安全所需的轮次数与丢弃安全机器的概率之间的权衡。接着，我们考虑在几乎必然约束下为马尔可夫决策过程（MDP）寻找最优策略的问题。我们证明动作值函数满足一种基于障碍的分解，该分解允许独立于奖励过程识别可行策略。利用这一分解，我们开发了一种障碍学习算法，该算法能在有限期望步数内识别出这类不安全的状态-动作对。我们的分析进一步凸显了底层MDP检测不安全行为所需的时间延迟与不安全事件暴露程度之间的权衡。模拟实验证实了我们的理论发现，进一步阐明了上述权衡，并表明安全约束可以加速学习过程。