As innovation in deep learning continues, many engineers seek to adopt Pre-Trained Models (PTMs) as components in computer systems. Researchers publish PTMs, which engineers adapt for quality or performance prior to deployment. PTM authors should choose appropriate names for their PTMs, which would facilitate model discovery and reuse. However, prior research has reported that model names are not always well chosen - and are sometimes erroneous. The naming for PTM packages has not been systematically studied. In this paper, we frame and conduct the first empirical investigation of PTM naming practices in the Hugging Face PTM registry. We initiated our study with a survey of 108 Hugging Face users to understand the practices in PTM naming. From our survey analysis, we highlight discrepancies from traditional software package naming, and present findings on naming practices. Our findings indicate there is a great mismatch between engineers' preferences and practical practices of PTM naming. We also present practices on detecting naming anomalies and introduce a novel automated DNN ARchitecture Assessment technique (DARA), capable of detecting PTM naming anomalies. We envision future works on leveraging meta-features of PTMs to improve model reuse and trustworthiness.
翻译:随着深度学习领域的持续创新,许多工程师倾向于将预训练模型(PTM)作为组件集成到计算机系统中。研究人员发布PTM后,工程师在部署前会根据质量或性能需求对其进行适配。PTM作者应为模型选取恰当的名称,以促进模型发现与复用。然而既有研究表明,模型命名并非总是合理——有时甚至存在错误。目前学术界尚未系统性地研究PTM包的命名实践。本文首次对Hugging Face PTM注册中心中的命名实践进行界定与实证探究。我们首先通过调查108位Hugging Face用户来理解PTM命名实践。基于调查分析,我们揭示了PTM命名与传统软件包命名的差异,并提出命名实践相关发现。研究结果表明,工程师的命名偏好与实际命名实践之间存在显著错位。我们还提出了检测命名异常的方法,并引入了一种新颖的自动化DNN架构评估技术(DARA),该技术能够检测PTM命名异常。我们展望未来工作可利用PTM元特征来提升模型复用性与可信度。