When deploying machine learning models in high-stakes robotics applications, the ability to detect unsafe situations is crucial. Early warning systems can provide alerts when an unsafe situation is imminent (in the absence of corrective action). To reliably improve safety, these warning systems should have a provable false negative rate; i.e. of the situations that are unsafe, fewer than $\epsilon$ will occur without an alert. In this work, we present a framework that combines a statistical inference technique known as conformal prediction with a simulator of robot/environment dynamics, in order to tune warning systems to provably achieve an $\epsilon$ false negative rate using as few as $1/\epsilon$ data points. We apply our framework to a driver warning system and a robotic grasping application, and empirically demonstrate guaranteed false negative rate while also observing low false detection (positive) rate.

翻译：在将机器学习模型部署于高风险的机器人应用场景时，检测不安全情况的能力至关重要。预警系统可在不安全情况即将发生时（若不采取纠正措施）发出警报。为切实提升安全性，此类预警系统应具备可证明的漏报率——即在不安全情况中，未触发警报的事件占比低于$\epsilon$。本研究提出一种框架，将统计推断技术"保形预测"与机器人/环境动态模拟器相结合，通过仅需$1/\epsilon$个数据点即可将预警系统调校至可证明的$\epsilon$漏报率。我们将该框架应用于驾驶员预警系统与机器人抓取任务，实验证明在实现低误报率的同时，其漏报率指标具备可证伪性。