Many engineering organizations are reimplementing and extending deep neural networks from the research community. We describe this process as deep learning model reengineering. Deep learning model reengineering - reusing, reproducing, adapting, and enhancing state-of-the-art deep learning approaches - is challenging for reasons including under-documented reference models, changing requirements, and the cost of implementation and testing. In addition, individual engineers may lack expertise in software engineering, yet teams must apply knowledge of software engineering and deep learning to succeed. Prior work has examined on DL systems from a "product" view, examining defects from projects regardless of the engineers' purpose. Our study is focused on reengineering activities from a "process" view, and focuses on engineers specifically engaged in the reengineering process. Our goal is to understand the characteristics and challenges of deep learning model reengineering. We conducted a case study of this phenomenon, focusing on the context of computer vision. Our results draw from two data sources: defects reported in open-source reeengineering projects, and interviews conducted with open-source project contributors and the leaders of a reengineering team. Our results describe how deep learning-based computer vision techniques are reengineered, analyze the distribution of defects in this process, and discuss challenges and practices. Integrating our quantitative and qualitative data, we proposed a novel reengineering workflow. Our findings inform several future directions, including: measuring additional unknown aspects of model reengineering; standardizing engineering practices to facilitate reengineering; and developing tools to support model reengineering and model reuse.
翻译:许多工程组织正在从研究社区重新实现并扩展深度神经网络。我们将这一过程定义为深度学习模型复刻。深度学习模型复刻——重用、复现、适配和增强前沿深度学习方法——面临诸多挑战,原因包括参考模型文档不足、需求变更以及实现与测试的成本。此外,单个工程师可能缺乏软件工程专业知识,而团队必须结合软件工程与深度学习知识才能成功。先前的研究从“产品”视角审视深度学习系统,检查项目中的缺陷而不考虑工程师的意图。我们的研究则聚焦于复刻活动,从“过程”视角出发,专门关注参与复刻流程的工程师。我们的目标是理解深度学习模型复刻的特征与挑战。我们以计算机视觉为背景,对此现象开展案例研究。结果来源于两个数据源:开源复刻项目中报告的缺陷,以及对开源项目贡献者和复刻团队领导者的访谈。我们的结果描述了基于深度学习的计算机视觉技术如何被复刻,分析了这一过程中的缺陷分布,并讨论了挑战与实践。通过整合定量与定性数据,我们提出了一种新颖的复刻工作流。我们的研究为多个未来方向提供了启示,包括:度量模型复刻中其他未知方面;标准化工程实践以促进复刻;以及开发支持模型复刻与模型重用的工具。