Software engineers develop, fine-tune, and deploy deep learning (DL) models using a variety of development frameworks and runtime environments. DL model converters move models between frameworks and to runtime environments. Conversion errors compromise model quality and disrupt deployment. However, the failure characteristics of DL model converters are unknown, adding risk when using DL interoperability technologies. This paper analyzes failures in DL model converters. We survey software engineers about DL interoperability tools, use cases, and pain points (N=92). Then, we characterize failures in model converters associated with the main interoperability tool, ONNX (N=200 issues in PyTorch and TensorFlow). Finally, we formulate and test two hypotheses about structural causes for the failures we studied. We find that the node conversion stage of a model converter accounts for ~75% of the defects and 33% of reported failure are related to semantically incorrect models. The cause of semantically incorrect models is elusive, but models with behaviour inconsistencies share operator sequences. Our results motivate future research on making DL interoperability software simpler to maintain, extend, and validate. Research into behavioural tolerances and architectural coverage metrics could be fruitful.

翻译：软件工程师使用多种开发框架和运行时环境来开发、微调和部署深度学习模型。深度学习模型转换器负责在不同框架之间以及向运行时环境迁移模型。转换错误会损害模型质量并中断部署进程。然而，深度学习模型转换器的故障特征尚不明确，这增加了使用深度学习互操作性技术的风险。本文对深度学习模型转换器的故障进行分析。我们通过问卷调查（N=92）了解软件工程师对深度学习互操作工具、应用场景及痛点的看法。随后，我们以主要互操作工具ONNX为对象，系统刻画了其相关模型转换器的故障特征（涵盖PyTorch和TensorFlow中的200个问题）。最后，我们针对所研究故障的结构性成因提出并验证了两个假设。研究发现：模型转换器的节点转换阶段贡献了约75%的缺陷，33%的已报告故障与语义错误的模型相关。语义错误模型的成因难以追溯，但存在行为不一致性的模型共享相同的算子序列。本研究结果启示未来应致力于简化深度学习互操作软件的维护、扩展与验证工作，同时探索行为容限与架构覆盖度指标的研究方向具有重要价值。