When we rely on deep-learned models for robotic perception, we must recognize that these models may behave unreliably on inputs dissimilar from the training data, compromising the closed-loop system's safety. This raises fundamental questions on how we can assess confidence in perception systems and to what extent we can take safety-preserving actions when external environmental changes degrade our perception model's performance. Therefore, we present a framework to certify the safety of a perception-enabled system deployed in novel contexts. To do so, we leverage robust model predictive control (MPC) to control the system using the perception estimates while maintaining the feasibility of a safety-preserving fallback plan that does not rely on the perception system. In addition, we calibrate a runtime monitor using recently proposed conformal prediction techniques to certifiably detect when the perception system degrades beyond the tolerance of the MPC controller, resulting in an end-to-end safety assurance. We show that this control framework and calibration technique allows us to certify the system's safety with orders of magnitudes fewer samples than required to retrain the perception network when we deploy in a novel context on a photo-realistic aircraft taxiing simulator. Furthermore, we illustrate the safety-preserving behavior of the MPC on simulated examples of a quadrotor. We open-source our simulation platform and provide videos of our results at our project page: https://tinyurl.com/fallback-safe-mpc.

翻译：当依赖深度学习模型进行机器人感知时，我们必须认识到这些模型可能在输入与训练数据差异较大时表现不可靠，从而危及闭环系统的安全性。这引发了两个根本性问题：如何评估感知系统的置信度，以及在外部环境变化导致感知模型性能退化时，能在多大程度上采取安全保护措施。为此，我们提出一个框架，用于验证部署在新场景下的感知系统安全性。该框架利用鲁棒模型预测控制（MPC）通过感知估计值控制系统，同时保持不依赖感知系统的安全保护备用方案的可行性。此外，我们采用近期提出的保形预测技术校准运行时监控器，以可验证的方式检测感知系统性能是否超出MPC控制器的容限阈值，从而形成端到端的安全保障。研究表明，在照片级飞机滑行模拟器的新场景部署条件下，该控制框架与校准技术仅需重新训练感知网络所需样本数量级的样本即可完成系统安全验证。我们进一步通过四旋翼飞行器的仿真示例展示了MPC的安全保护行为。相关仿真平台已开源，完整结果视频可通过项目页面获取：https://tinyurl.com/fallback-safe-mpc