Replication packages are crucial for enabling transparency, validation, and reuse in software engineering (SE) research. While artifact sharing is now a standard practice and even expected at premier SE venues such as ICSE, the practical usability of these replication packages remains underexplored. In particular, there is a marked lack of studies that comprehensively examine the executability and reproducibility of replication packages in SE research. In this paper, we aim to fill this gap by evaluating 100 replication packages published as part of ICSE proceedings over the past decade (2015--2024). We assess the (1) executability of the replication packages, (2) efforts and modifications required to execute them, (3) challenges that prevent executability, and (4) reproducibility of the original findings. We spent approximately 650 person-hours in total executing the artifacts and reproducing the study findings. Our findings reveal that only 40\% of the 100 evaluated artifacts were executable, of which 32.5\% (13 out of 40) ran without any modification. Regarding effort levels, 17.5\% (7 out of 40) required low effort, while 82.5\% (33 out of 40) required moderate to high effort to execute successfully. We identified five common types of modifications and 13 challenges leading to execution failure, spanning environmental, documentation, and structural issues. Among the executable artifacts, only 35\% (14 out of 40) reproduced the original results. These findings highlight a notable gap between artifact availability, executability, and reproducibility. Our study proposes three actionable guidelines to improve the preparation, documentation, and review of research artifacts, thereby strengthening the rigor and sustainability of open science practices in SE research.

翻译：在软件工程研究中，复制包对于实现透明度、验证和重用至关重要。尽管在ICSE等顶级软件工程会议上，共享研究制品已成为标准实践甚至被普遍期待，但这些复制包的实际可用性仍未得到充分探索。特别是，目前明显缺乏对软件工程研究中复制包的可执行性和可复现性进行全面考察的研究。本文旨在填补这一空白，通过评估过去十年间作为ICSE会议录组成部分发表的100个复制包。我们评估了这些复制包的（1）可执行性，（2）执行所需的工作量和修改，（3）导致无法执行的挑战，以及（4）原始结果的可复现性。我们总计花费了约650人时来执行这些制品并复现研究结果。我们的发现表明，在评估的100个制品中，仅有40%是可执行的，其中32.5%（40个中的13个）无需任何修改即可运行。在工作量方面，17.5%（40个中的7个）需要低工作量，而82.5%（40个中的33个）需要中等到高工作量才能成功执行。我们识别了五种常见的修改类型和13种导致执行失败的挑战，涵盖环境、文档和结构性问题。在可执行的制品中，仅有35%（40个中的14个）复现了原始结果。这些发现凸显了制品可用性、可执行性和可复现性之间存在显著差距。本研究提出了三条可操作的指导原则，以改进研究制品的准备、文档和评审，从而增强软件工程研究中开放科学实践的严谨性和可持续性。