Image recognition tasks typically use deep learning and require enormous processing power, thus relying on hardware accelerators like GPUs and TPUs for fast, timely processing. Failure in real-time image recognition tasks can occur due to sub-optimal mapping on hardware accelerators during model deployment, which may lead to timing uncertainty and erroneous behavior. Mapping on hardware accelerators is done using multiple software components like deep learning frameworks, compilers, and device libraries, that we refer to as the computational environment. Owing to the increased use of image recognition tasks in safety-critical applications like autonomous driving and medical imaging, it is imperative to assess their robustness to changes in the computational environment, as the impact of parameters like deep learning frameworks, compiler optimizations, and hardware devices on model performance and correctness is not yet well understood. In this paper we present a differential testing framework, DeltaNN, that allows us to assess the impact of different computational environment parameters on the performance of image recognition models during deployment, post training. DeltaNN generates different implementations of a given image recognition model for variations in environment parameters, namely, deep learning frameworks, compiler optimizations and hardware devices and analyzes differences in model performance as a result. Using DeltaNN, we conduct an empirical study of robustness analysis of three popular image recognition models using the ImageNet dataset. We report the impact in terms of misclassifications and inference time differences across different settings. In total, we observed up to 72% output label differences across deep learning frameworks, and up to 81% unexpected performance degradation in terms of inference time, when applying compiler optimizations.

翻译：图像识别任务通常采用深度学习技术，需要巨大的计算能力，因而依赖GPU和TPU等硬件加速器以实现快速及时的处理。在模型部署过程中，由于硬件加速器上的次优映射可能导致时序不确定性和错误行为，实时图像识别任务可能失败。硬件加速器上的映射通过多个软件组件（如深度学习框架、编译器和设备库）完成，我们将这些组件统称为计算环境。随着图像识别任务在自动驾驶和医学成像等安全关键型应用中的日益普及，评估其对计算环境变化的鲁棒性变得至关重要，因为深度学习框架、编译器优化和硬件设备等参数对模型性能和正确性的影响尚未得到充分理解。本文提出了一种差分测试框架DeltaNN，用于评估不同计算环境参数对部署后（训练后）图像识别模型性能的影响。DeltaNN针对环境参数（即深度学习框架、编译器优化和硬件设备）的变化，生成给定图像识别模型的不同实现，并分析由此产生的模型性能差异。利用DeltaNN，我们使用ImageNet数据集对三种主流图像识别模型进行了鲁棒性分析的实证研究。我们报告了不同设置下错误分类和推理时间差异方面的影响。总体而言，我们观察到不同深度学习框架间的输出标签差异高达72%，而应用编译器优化时推理时间的意外性能下降高达81%。