Many software engineers develop, fine-tune, and deploy deep learning (DL) models. They use DL models in a variety of development frameworks and deploy to a range of runtime environments. In this diverse ecosystem, engineers use DL model converters to move models from frameworks to runtime environments. Conversion errors compromise model quality and disrupt deployment. However, failure modes and patterns of DL model converters are unknown. This knowledge gap adds engineering risk in DL interoperability technologies. In this paper, we conduct the first failure analysis on DL model converters. Specifically, we characterize failures in model converters associated with ONNX (Open Neural Network eXchange). We analyze failures in the ONNX converters for two major DL frameworks, PyTorch and TensorFlow. The symptoms, causes, and locations of failures are reported for N=200 issues. We also evaluate why models fail by converting 5,149 models, both real-world and synthetically generated instances. Through the course of our testing, we find 11 defects (5 new) across torch.onnx, tf2onnx, and the ONNXRuntime. We evaluated two hypotheses about the relationship between model operators and converter failures, falsifying one and with equivocal results on the other. We describe and note weaknesses in the current testing strategies for model converters. Our results motivate future research on making DL software simpler to maintain, extend, and validate.

翻译：众多软件工程师在开发、微调和部署深度学习模型时，会使用多种开发框架，并将其部署到各类运行时环境中。在这个多元化的生态中，工程师借助深度学习模型转换器，将模型从开发框架迁移至运行时环境。然而，转换错误会损害模型质量并中断部署流程。目前，深度学习模型转换器的故障模式与规律尚不明确，这一知识空白增加了深度学习互操作技术中的工程风险。本文首次开展了深度学习模型转换器的故障分析研究。具体而言，我们针对与ONNX（开放神经网络交换格式）相关的模型转换器中的故障进行了系统表征。我们分析了PyTorch和TensorFlow两大主流深度学习框架的ONNX转换器中的故障，并记录了N=200个问题的症状、原因及故障位置。同时，通过转换5,149个真实世界与合成生成的模型实例，探究了模型失败的原因。测试过程中，我们在torch.onnx、tf2onnx和ONNXRuntime中发现了11个缺陷（其中5个为新发现）。我们针对模型算子与转换器故障之间的关联提出了两个假设，经检验否定了其中一个假设，而另一个假设的检验结果尚不明确。本文还指出了当前模型转换器测试策略的局限性，并强调了简化深度学习软件维护、扩展与验证流程的未来研究方向。