Testing Deep Learning (DL)-based systems is an open challenge. Although it is relatively easy to find inputs that cause a DL model to misbehave, the grouping of inputs by features that make the DL model under test fail is largely unexplored. Existing approaches for DL testing introduce perturbations that may focus on specific failure-inducing features, while neglecting others that belong to different regions of the feature space. In this paper, we create an explicit topographical map of the input feature space. Our approach, named TopoMap, is both black-box and model-agnostic as it relies solely on features that characterise the input space. To discriminate the inputs according to the specific features they share, we first apply dimensionality reduction to obtain input embeddings, which are then subjected to clustering. Each DL model might require specific embedding computations and clustering algorithms to achieve a meaningful separation of inputs into discriminative groups. We propose a novel way to evaluate alternative configurations of embedding and clustering techniques. We used a deep neural network (DNN) as an approximation of a human evaluator who could tell whether a pair of clusters can be discriminated based on the features of the included elements. We use such a DNN to automatically select the optimal topographical map of the inputs among all those that are produced by different embedding/clustering configurations. The evaluation results show that the maps generated by TopoMap consist of distinguishable and meaningful regions. In addition, we evaluate the effectiveness of TopoMap using mutation analysis. In particular, we assess whether the clusters in our topographical map allow for an effective selection of mutation-killing inputs. Experimental results show that our approach outperforms random selection by 35% on average on killable mutants; by 61% on non-killable ones.

翻译：测试基于深度学习系统的可靠性是一项开放性挑战。尽管找到导致模型异常行为的输入相对容易，但根据引发模型失效的特征对输入进行分组尚鲜有研究。现有深度学习测试方法引入的扰动可能聚焦于特定失效诱因特征，却忽略了特征空间中其他区域的特征。本文构建了输入特征空间的显式地形图。我们提出的TopoMap方法兼具黑盒特性与模型无关性，仅依赖表征输入空间的特征。为根据输入共享的特征进行判别，我们首先通过降维获取输入嵌入表示，再对其进行聚类。不同深度学习模型可能需要特定的嵌入计算与聚类算法以实现有意义的输入群组划分。我们提出了一种评估嵌入与聚类技术组合配置的新方法：采用深度神经网络模拟人类评估者，由其判别聚类对是否可根据所含元素的特征加以区分。通过该深度神经网络，我们从不同嵌入/聚类配置产生的候选图中自动选取最优输入地形图。评估结果表明，TopoMap生成的地图由可区分且有意义的区域构成。此外，我们通过变异分析评估了TopoMap的有效性：具体而言，验证地形图中的聚类是否支持有效选取可杀死变体的输入。实验结果显示，我们的方法在可杀死变体上平均优于随机选择35%，在不可杀死变体上提升61%。