Due to the black box nature of Convolutional Neural Networks (CNNs), the continuous validation of CNNs during operation is challenging with the absence of a human monitor. As a result this makes it difficult for developers and regulators to gain confidence in the deployment of autonomous systems employing CNNs. It is critical for safety during operation to know when CNN's predictions are trustworthy or suspicious. With the absence of a human monitor, the basic approach is to use the model's output confidence score to assess if predictions are trustworthy or suspicious. However, the model's confidence score is a result of computations coming from a black box, therefore lacks transparency and makes it challenging to automatedly credit trustworthiness to predictions. We introduce the trustworthiness score (TS), a simple metric that provides a more transparent and effective way of providing confidence in CNNs predictions compared to model's confidence score. The metric quantifies the trustworthiness in a prediction by checking for the existence of certain features in the predictions made by the CNN. We also use the underlying idea of the TS metric, to provide a suspiciousness score (SS) in the overall input frame to help in the detection of suspicious frames where false negatives exist. We conduct a case study using YOLOv5 on persons detection to demonstrate our method and usage of TS and SS. The case study shows that using our method consistently improves the precision of predictions compared to relying on model confidence score alone, for both 1) approving of trustworthy predictions (~20% improvement) and 2) detecting suspicious frames (~5% improvement).

翻译：由于卷积神经网络（CNNs）的黑箱特性，在缺乏人工监控的情况下，运行期间持续验证CNNs面临挑战。这使得开发者和监管者难以对采用CNNs的自主系统部署建立信心。了解CNN预测何时可信或可疑，对运行安全至关重要。在无人工监控时，基本方法是使用模型输出的置信度分数来评估预测的可信或可疑程度。然而，模型置信度分数来自黑箱计算，缺乏透明度，难以自动为预测赋予可信度。我们引入信任分数（TS）这一简单指标，与模型置信度分数相比，它能以更透明有效的方式提供对CNN预测的置信度。该指标通过检查CNN预测中是否存在特定特征来量化预测的可信度。同时，利用TS指标的核心思想，我们提供整体输入帧的可疑度分数（SS），协助检测存在假阴性的可疑帧。我们以YOLOv5行人检测为例进行案例研究，展示TS和SS方法的应用。案例研究表明，与仅依赖模型置信度分数相比，使用我们的方法在以下两方面均能持续提升预测精度：1）确认可信预测（提升约20%）；2）检测可疑帧（提升约5%）。