Due to the black box nature of Convolutional Neural Networks (CNNs), the continuous validation of CNNs during operation is challenging with the absence of a human monitor. As a result this makes it difficult for developers and regulators to gain confidence in the deployment of autonomous systems employing CNNs. It is critical for safety during operation to know when CNN's predictions are trustworthy or suspicious. With the absence of a human monitor, the basic approach is to use the model's output confidence score to assess if predictions are trustworthy or suspicious. However, the model's confidence score is a result of computations coming from a black box, therefore lacks transparency and makes it challenging to automatedly credit trustworthiness to predictions. We introduce the trustworthiness score (TS), a simple metric that provides a more transparent and effective way of providing confidence in CNNs predictions compared to model's confidence score. The metric quantifies the trustworthiness in a prediction by checking for the existence of certain features in the predictions made by the CNN. We also use the underlying idea of the TS metric, to provide a suspiciousness score (SS) in the overall input frame to help in the detection of suspicious frames where false negatives exist. We conduct a case study using YOLOv5 on persons detection to demonstrate our method and usage of TS and SS. The case study shows that using our method consistently improves the precision of predictions compared to relying on model confidence score alone, for both 1) approving of trustworthy predictions (~20% improvement) and 2) detecting suspicious frames (~5% improvement).

翻译：由于卷积神经网络（CNNs）的黑箱特性，在缺乏人工监控的情况下，运行期间持续验证CNNs存在困难。这使得开发人员和监管机构难以对采用CNNs的自主系统部署建立信心。运行安全的关键在于判断CNN的预测结果是否可信或可疑。在缺乏人工监控时，基本方法是利用模型输出的置信度分数评估预测的可信度。然而，模型的置信度分数源于黑箱计算过程，缺乏透明度，难以自动为预测结果赋予可信度。我们提出可信度分数（TS）这一简单指标，相比模型置信度分数，它能以更透明的方式有效建立对CNN预测的置信度。该指标通过检测CNN预测中特定特征的存在与否量化预测可信度。同时利用TS指标的核心思想，提出整体输入帧的可疑度分数（SS），辅助检测存在假阴性异常的可疑帧。我们以YOLOv5行人检测为例进行案例研究，展示TS和SS的使用方法。案例研究表明：相较于仅依赖模型置信度分数，采用我们的方法在两类任务中均能持续提升预测精度——1）可信预测的批准（提升约20%）；2）可疑帧检测（提升约5%）。