Due to the black box nature of deep neural networks (DNN), the continuous validation of DNN during operation is challenging with the absence of a human monitor. As a result this makes it difficult for developers and regulators to gain confidence in the deployment of autonomous systems employing DNN. It is critical for safety during operation to know when DNN's predictions are trustworthy or suspicious. With the absence of a human monitor, the basic approach is to use the model's output confidence score to assess if predictions are trustworthy or suspicious. However, the model's confidence score is a result of computations coming from a black box, therefore lacks transparency and makes it challenging to automatedly credit trustworthiness to predictions. We introduce the trustworthiness score (TS), a simple metric that provides a more transparent and effective way of providing confidence in DNN predictions compared to model's confidence score. The metric quantifies the trustworthiness in a prediction by checking for the existence of certain features in the predictions made by the DNN. We also use the underlying idea of the TS metric, to provide a suspiciousness score (SS) in the overall input frame to help in the detection of suspicious frames where false negatives exist. We conduct a case study using YOLOv5 on persons detection to demonstrate our method and usage of TS and SS. The case study shows that using our method consistently improves the precision of predictions compared to relying on model confidence score alone, for both 1) approving of trustworthy predictions (~20% improvement) and 2) detecting suspicious frames (~5% improvement).

翻译：由于深度神经网络（DNN）的黑箱特性，在运行时缺乏人工监控的情况下，持续验证DNN性能具有挑战性。这使得开发者和监管机构难以对采用DNN的自主系统部署建立信心。在运行过程中，判断DNN预测是可信还是可疑对安全保障至关重要。在缺乏人工监控时，基本方法是使用模型输出的置信度评分来评估预测的可信度或可疑程度。然而，模型置信度评分源自黑箱计算过程，缺乏透明度，难以自动化地将可信度赋予预测结果。我们引入可信度评分（TS）这一简单指标，相比模型置信度评分，它能以更透明、更有效的方式衡量DNN预测的可信度。该指标通过检查DNN预测中是否存在特定特征来量化预测的可信度。同时，我们利用TS指标的核心理念，为整体输入帧提供可疑度评分（SS），以辅助检测存在假阴性结果的异常帧。我们以YOLOv5在行人检测中的应用为例开展案例研究，验证TS和SS方法及使用方式。案例研究表明，相比单独依赖模型置信度评分，使用我们的方法在以下两方面一致性地提升了预测精度：1）可信预测的确认（提升约20%），2）异常帧的检测（提升约5%）。