The behavior of neural networks (NNs) on previously unseen types of data (out-of-distribution or OOD) is typically unpredictable. This can be dangerous if the network's output is used for decision-making in a safety-critical system. Hence, detecting that an input is OOD is crucial for the safe application of the NN. Verification approaches do not scale to practical NNs, making runtime monitoring more appealing for practical use. While various monitors have been suggested recently, their optimization for a given problem, as well as comparison with each other and reproduction of results, remain challenging. We present a tool for users and developers of NN monitors. It allows for (i) application of various types of monitors from the literature to a given input NN, (ii) optimization of the monitor's hyperparameters, and (iii) experimental evaluation and comparison to other approaches. Besides, it facilitates the development of new monitoring approaches. We demonstrate the tool's usability on several use cases of different types of users as well as on a case study comparing different approaches from recent literature.

翻译：神经网络（NN）对先前未见数据类型（分布外或OOD）的行为通常是不可预测的。若网络输出用于安全关键系统的决策制定，这可能带来危险。因此，检测输入是否为OOD对于神经网络的安全应用至关重要。验证方法无法扩展到实际神经网络，使得运行时监测在实践中更具吸引力。尽管近期提出了多种监测器，但针对特定问题的优化、相互比较以及结果的复现仍具有挑战性。我们提出了一款面向神经网络监测器用户与开发者的工具。该工具支持：（i）将文献中的多种监测器类型应用于给定输入神经网络，（ii）优化监测器的超参数，以及（iii）实验评估并与其他方法进行比较。此外，它还有助于开发新的监测方法。我们通过多个不同用户类型的用例以及一项对比近期文献中不同方法的案例研究，展示了该工具的实用性。