Deep neural networks (DNNs) are widely used in various application domains such as image processing, speech recognition, and natural language processing. However, testing DNN models may be challenging due to the complexity and size of their input domain. Particularly, testing DNN models often requires generating or exploring large unlabeled datasets. In practice, DNN test oracles, which identify the correct outputs for inputs, often require expensive manual effort to label test data, possibly involving multiple experts to ensure labeling correctness. In this paper, we propose DeepGD, a black-box multi-objective test selection approach for DNN models. It reduces the cost of labeling by prioritizing the selection of test inputs with high fault revealing power from large unlabeled datasets. DeepGD not only selects test inputs with high uncertainty scores to trigger as many mispredicted inputs as possible but also maximizes the probability of revealing distinct faults in the DNN model by selecting diverse mispredicted inputs. The experimental results conducted on four widely used datasets and five DNN models show that in terms of fault-revealing ability: (1) White-box, coverage-based approaches fare poorly, (2) DeepGD outperforms existing black-box test selection approaches in terms of fault detection, and (3) DeepGD also leads to better guidance for DNN model retraining when using selected inputs to augment the training set.

翻译：深度神经网络（DNN）已广泛应用于图像处理、语音识别和自然语言处理等多个应用领域。然而，由于DNN模型输入域的复杂性和规模，对其进行测试可能面临挑战。具体而言，测试DNN模型通常需要生成或探索大量未标注数据集。在实际应用中，用于识别输入正确输出的DNN测试预言器往往需要耗费大量人工成本来标注测试数据，甚至可能涉及多位专家以确保标注正确性。本文提出DeepGD——一种面向DNN模型的黑盒多目标测试选择方法。该方法通过从大规模未标注数据集中优先选择故障发现能力强的测试输入，从而降低标注成本。DeepGD不仅通过选择高不确定性得分的测试输入来尽可能触发更多误预测样本，还通过选择多样化的误预测输入来最大化揭示DNN模型中不同故障的概率。在四个广泛使用的数据集和五个DNN模型上开展的实验结果表明，就故障发现能力而言：（1）基于覆盖率的白盒方法表现欠佳；（2）DeepGD在故障检测方面优于现有黑盒测试选择方法；（3）当使用所选输入增强训练集时，DeepGD能为DNN模型重训练提供更优的指导。