Software engineers develop, fine-tune, and deploy deep learning (DL) models. They use and re-use models in a variety of development frameworks and deploy them on a range of runtime environments. In this diverse ecosystem, engineers use DL model converters to move models from frameworks to runtime environments. However, errors in converters can compromise model quality and disrupt deployment. The failure frequency and failure modes of DL model converters are unknown. In this paper, we conduct the first failure analysis on DL model converters. Specifically, we characterize failures in model converters associated with ONNX (Open Neural Network eXchange). We analyze past failures in the ONNX converters in two major DL frameworks, PyTorch and TensorFlow. The symptoms, causes, and locations of failures (for N=200 issues), and trends over time are also reported. We also evaluate present-day failures by converting 8,797 models, both real-world and synthetically generated instances. The consistent result from both parts of the study is that DL model converters commonly fail by producing models that exhibit incorrect behavior: 33% of past failures and 8% of converted models fell into this category. Our results motivate future research on making DL software simpler to maintain, extend, and validate.

翻译：软件工程师开发、微调并部署深度学习（DL）模型。他们在多种开发框架中使用和复用模型，并将其部署于一系列运行时环境中。在这一多元化生态系统中，工程师利用DL模型转换器将模型从框架迁移至运行时环境。然而，转换器中的错误可能损害模型质量并中断部署过程。目前DL模型转换器的故障频率与故障模式尚不明确。本文首次对DL模型转换器进行故障分析，具体而言，我们刻画了与ONNX（开放神经网络交换格式）相关的模型转换器中的故障特征。我们分析了PyTorch和TensorFlow两大主流DL框架中ONNX转换器的历史故障，报告了故障的症状、原因、位置（共N=200个问题）及其随时间变化的趋势。同时，通过转换8797个模型（包括真实世界与合成生成的实例）评估当下存在的故障。两项研究得出的一致结论是：DL模型转换器的常见故障表现为生成行为异常的模型——33%的历史故障与8%的转换后模型属于此类。本研究结果将推动未来关于提升DL软件维护、扩展与验证简便性的相关研究。