Testing black-box perceptual-control systems in simulation faces two difficulties. Firstly, perceptual inputs in simulation lack the fidelity of real-world sensor inputs. Secondly, for a reasonably accurate perception system, encountering a rare failure trajectory may require running infeasibly many simulations. This paper combines perception error models -- surrogates for a sensor-based detection system -- with state-dependent adaptive importance sampling. This allows us to efficiently assess the rare failure probabilities for real-world perceptual control systems within simulation. Our experiments with an autonomous braking system equipped with an RGB obstacle-detector show that our method can calculate accurate failure probabilities with an inexpensive number of simulations. Further, we show how choice of safety metric can influence the process of learning proposal distributions capable of reliably sampling high-probability failures.

翻译：在仿真中测试黑盒感知-控制系统面临两个困难。首先，仿真中的感知输入缺乏真实世界传感器输入的保真度。其次，对于精度合理的感知系统而言，遇到罕见的故障轨迹可能需要运行数量过多而不可行的仿真。本文结合感知误差模型——基于传感器的检测系统的替代模型——与状态相关的自适应重要性采样。这使得我们能够在仿真中高效评估真实世界感知控制系统的罕见故障概率。实验表明，配备RGB障碍物检测器的自主制动系统能够以较低的仿真次数计算出精确的故障概率。此外，我们展示了安全度量的选择如何影响学习能够可靠采样高概率故障的建议分布的过程。