Runtime assurance (RTA) addresses the problem of keeping an autonomous system safe while using an untrusted (or experimental) controller. This can be done via logic that explicitly switches between the untrusted controller and a safety controller, or logic that filters the input provided by the untrusted controller. While several tools implement specific instances of RTAs, there is currently no framework for evaluating different approaches. Given the importance of the RTA problem in building safe autonomous systems, an evaluation tool is needed. In this paper, we present the RTAEval framework as a low code framework that can be used to quickly evaluate different RTA logics for different types of agents in a variety of scenarios. RTAEval is designed to quickly create scenarios, run different RTA logics, and collect data that can be used to evaluate and visualize performance. In this paper, we describe different components of RTAEval and show how it can be used to create and evaluate scenarios involving multiple aircraft models.

翻译：运行时保证（RTA）解决了在使用不可信（或实验性）控制器时保持自主系统安全的问题。这可以通过在不可信控制器与安全控制器之间显式切换的逻辑，或者过滤不可信控制器输入的逻辑来实现。尽管已有多种工具实现了特定的RTA实例，但目前缺乏一个用于评估不同方法的框架。鉴于RTA问题在构建安全自主系统中的重要性，亟需一个评估工具。本文提出了RTAEval框架，作为低代码框架，可用于快速评估不同代理类型在多种场景下的不同RTA逻辑。RTAEval旨在快速创建场景、运行不同RTA逻辑，并收集可用于评估与可视化性能的数据。本文描述了RTAEval的不同组件，并展示了如何将其用于创建和评估涉及多种飞机模型的场景。