The Software Engineering (SE) community has been embracing the open science policy and encouraging researchers to disclose artifacts in their publications. However, the status and trends of artifact practice and quality remain unclear, lacking insights on further improvement. In this paper, we present an empirical study to characterize the research artifacts in SE publications. Specifically, we manually collect 1,487 artifacts from all 2,196 papers published in top-tier SE conferences (ASE, FSE, ICSE, and ISSTA) from 2017 to 2022. We investigate the common practices (e.g., URL location and format, storage websites), maintenance activities (e.g., last update time and URL validity), popularity (e.g., the number of stars on GitHub and characteristics), and quality (e.g., documentation and code smell) of these artifacts. Based on our analysis, we reveal a rise in publications providing artifacts. The usage of Zenodo for sharing artifacts has significantly increased. However, artifacts stored in GitHub tend to receive few stars, indicating a limited influence on real-world SE applications. We summarize the results and provide suggestions to different stakeholders in conjunction with current guidelines.
翻译:软件工程(SE)社区已积极采纳开放科学政策,鼓励研究人员在其出版物中公开研究制品。然而,制品的实践现状与质量趋势仍不明确,缺乏对进一步改进的深入见解。本文通过实证研究,系统刻画了SE出版物中的研究制品特征。具体而言,我们手动收集了2017年至2022年间发表于顶级SE会议(ASE、FSE、ICSE、ISSTA)的2,196篇论文中的1,487个制品,并分别从常见实践(如URL位置与格式、存储网站)、维护活动(如最后更新时间与URL有效性)、流行度(如GitHub星标数及特征)以及质量(如文档完备性与代码异味)四个维度展开分析。研究显示,提供制品的论文数量呈增长趋势,其中Zenodo作为共享平台的使用率显著提升。然而,存储于GitHub的制品普遍星标数较低,表明其对现实SE应用的影响力有限。我们总结研究结果,并结合现有指南为不同利益相关方提出改进建议。