Estimating the probability of failure is an important step in the certification of safety-critical systems. Efficient estimation methods are often needed due to the challenges posed by high-dimensional input spaces, risky test scenarios, and computationally expensive simulators. This work frames the problem of black-box safety validation as a Bayesian optimization problem and introduces a method that iteratively fits a probabilistic surrogate model to efficiently predict failures. The algorithm is designed to search for failures, compute the most-likely failure, and estimate the failure probability over an operating domain using importance sampling. We introduce three acquisition functions that aim to reduce uncertainty by covering the design space, optimize the analytically derived failure boundaries, and sample the predicted failure regions. Results show this Bayesian safety validation approach provides a more accurate estimate of failure probability with orders of magnitude fewer samples and performs well across various safety validation metrics. We demonstrate this approach on three test problems, a stochastic decision making system, and a neural network-based runway detection system. This work is open sourced (https://github.com/sisl/BayesianSafetyValidation.jl) and currently being used to supplement the FAA certification process of the machine learning components for an autonomous cargo aircraft.

翻译：故障概率估计是安全关键系统认证中的重要环节。由于高维输入空间、高风险测试场景和计算成本高昂的仿真器带来的挑战，通常需要高效的估计方法。本研究将黑盒安全验证问题构建为贝叶斯优化问题，提出一种通过迭代拟合概率代理模型来高效预测故障的方法。该算法设计用于搜索故障、计算最可能故障，并利用重要性采样在工作域内估计故障概率。我们引入了三种采集函数，分别旨在通过覆盖设计空间来降低不确定性、优化解析推导的故障边界，以及对预测故障区域进行采样。结果表明，这种贝叶斯安全验证方法能够以数量级更少的样本获得更精确的故障概率估计，并在各类安全验证指标上表现优异。我们在三个测试问题、一个随机决策系统和一个基于神经网络的跑道检测系统上验证了该方法。本工作已开源（https://github.com/sisl/BayesianSafetyValidation.jl），目前正被用于补充自主货运飞机机器学习部件的FAA认证流程。