A package's source code repository records the development history of the package, providing indispensable information for the use and risk monitoring of the package. However, a package release often misses its source code repository due to the separation of the package's development platform from its distribution platform. Existing tools retrieve the release's repository information from its metadata, which suffers from two limitations: the metadata may not contain or contain wrong information. Our analysis shows that existing tools can only retrieve repository information for up to 70.5% of PyPI releases. To address the limitations, this paper proposes PyRadar, a novel framework that utilizes the metadata and source distribution to retrieve and validate the repository information for PyPI releases. We start with an empirical study to compare four existing tools on 4,227,425 PyPI releases and analyze phantom files (files appearing in the release's distribution but not in the release's repository) in 14,375 correct package-repository links and 2,064 incorrect links. Based on the findings, we design PyRadar with three components, i.e., Metadata-based Retriever, Source Code Repository Validator, and Source Code-based Retriever. In particular, the Metadata-based Retriever combines best practices of existing tools and successfully retrieves repository information from the metadata for 72.1% of PyPI releases. The Source Code Repository Validator applies common machine learning algorithms on six crafted features and achieves an AUC of up to 0.995. The Source Code-based Retriever queries World of Code with the SHA-1 hashes of all Python files in the release's source distribution and retrieves repository information for 90.2% of packages in our dataset with an accuracy of 0.970. Both practitioners and researchers can employ the PyRadar to better use PyPI packages.
翻译:包的源代码仓库记录了其开发历史,为包的使用和风险监控提供了不可或缺的信息。然而,由于包的开发平台与分发平台相分离,包的发行版本常常缺失其源代码仓库信息。现有工具从包的元数据中检索发行版本的仓库信息,但存在两个局限性:元数据可能缺失或包含错误信息。我们的分析表明,现有工具最多仅能检索到70.5%的PyPI发行版本的仓库信息。针对这些局限性,本文提出了PyRadar——一种新颖框架,通过利用元数据和源码分发包来检索并验证PyPI发行版本的仓库信息。我们首先开展了一项实证研究,在4,227,425个PyPI发行版本上对比了四种现有工具,并分析了14,375个正确包-仓库链接与2,064个错误链接中的幽灵文件(即出现在发行版分发包中但未出现在其仓库中的文件)。基于研究发现,我们设计了包含三个组件的PyRadar:基于元数据的检索器、源码仓库验证器以及基于源码的检索器。其中,基于元数据的检索器整合了现有工具的最佳实践,成功从元数据中检索到72.1%的PyPI发行版本的仓库信息。源码仓库验证器在六项精心设计的特征上应用常见机器学习算法,AUC值高达0.995。基于源码的检索器通过查询World of Code数据库,利用发行版源码分发包中所有Python文件的SHA-1哈希值进行检索,在我们数据集上以0.970的准确率成功检索到90.2%包的仓库信息。从业者和研究者均可借助PyRadar更好地使用PyPI包。