Accurately estimating the probability of failure for safety-critical systems is important for certification. Estimation is often challenging due to high-dimensional input spaces, dangerous test scenarios, and computationally expensive simulators; thus, efficient estimation techniques are important to study. This work reframes the problem of black-box safety validation as a Bayesian optimization problem and introduces an algorithm, Bayesian safety validation, that iteratively fits a probabilistic surrogate model to efficiently predict failures. The algorithm is designed to search for failures, compute the most-likely failure, and estimate the failure probability over an operating domain using importance sampling. We introduce a set of three acquisition functions that focus on reducing uncertainty by covering the design space, optimizing the analytically derived failure boundaries, and sampling the predicted failure regions. Mainly concerned with systems that only output a binary indication of failure, we show that our method also works well in cases where more output information is available. Results show that Bayesian safety validation achieves a better estimate of the probability of failure using orders of magnitude fewer samples and performs well across various safety validation metrics. We demonstrate the algorithm on three test problems with access to ground truth and on a real-world safety-critical subsystem common in autonomous flight: a neural network-based runway detection system. This work is open sourced and currently being used to supplement the FAA certification process of the machine learning components for an autonomous cargo aircraft.

翻译：准确评估安全关键系统的失效概率对于认证至关重要。由于高维输入空间、危险测试场景以及计算开销巨大的仿真器，该评估通常极具挑战性；因此，研究高效评估技术具有重要意义。本文将黑盒安全验证问题重新定义为贝叶斯优化问题，并提出一种名为"贝叶斯安全验证"的算法，该算法通过迭代拟合概率代理模型来高效预测失效。该算法旨在搜索失效模式、计算最可能失效场景，并利用重要性采样估计运行域上的失效概率。我们引入一组三个采集函数，分别侧重于通过覆盖设计空间降低不确定性、优化解析推导的失效边界以及采样预测的失效区域。方法主要针对仅输出二元失效指示的系统，同时证明其在可获得更多输出信息的场景下同样表现良好。结果表明，贝叶斯安全验证能以数量级更少的样本量获得更优的失效概率估计，并在多种安全验证指标上表现优异。我们在三个具有真实基准的测试问题以及自主飞行中常见的真实安全关键子系统（基于神经网络的跑道检测系统）上验证了该算法。本研究成果已开源，目前正用于辅助美国联邦航空管理局（FAA）对自主货运飞机机器学习组件的认证流程。