Replication packages are crucial for enabling transparency, validation, and reuse in software engineering (SE) research. While artifact sharing is now a standard practice and even expected at premier SE venues such as ICSE, the practical usability of these replication packages remains underexplored. In particular, there is a marked lack of studies that comprehensively examine the executability and reproducibility of replication packages in SE research. In this paper, we aim to fill this gap by evaluating 100 replication packages published as part of ICSE proceedings over the past decade (2015--2024). We assess the (1) executability of the replication packages, (2) efforts and modifications required to execute them, (3) challenges that prevent executability, and (4) reproducibility of the original findings. We spent approximately 650 person-hours in total executing the artifacts and reproducing the study findings. Our findings reveal that only 40\% of the 100 evaluated artifacts were executable, of which 32.5\% (13 out of 40) ran without any modification. Regarding effort levels, 17.5\% (7 out of 40) required low effort, while 82.5\% (33 out of 40) required moderate to high effort to execute successfully. We identified five common types of modifications and 13 challenges leading to execution failure, spanning environmental, documentation, and structural issues. Among the executable artifacts, only 35\% (14 out of 40) reproduced the original results. These findings highlight a notable gap between artifact availability, executability, and reproducibility. Our study proposes three actionable guidelines to improve the preparation, documentation, and review of research artifacts, thereby strengthening the rigor and sustainability of open science practices in SE research.

翻译：复现包对于实现软件工程（SE）研究的透明度、验证和重用至关重要。尽管在ICSE等顶级SE会议上，研究产物共享已成为标准实践甚至被普遍期待，但这些复现包的实际可用性仍未得到充分探索。特别是，目前明显缺乏全面考察SE研究中复现包可执行性与可复现性的研究。本文旨在通过评估过去十年（2015-2024年）作为ICSE会议录组成部分发表的100个复现包来填补这一空白。我们评估了：（1）复现包的可执行性；（2）执行所需的工作量与修改；（3）导致无法执行的挑战；（4）原始结果的可复现性。我们总计投入约650人时来执行这些研究产物并复现研究结果。我们的发现表明：在评估的100个研究产物中，仅有40%具备可执行性，其中32.5%（40个中的13个）无需任何修改即可运行。在工作量层面，17.5%（40个中的7个）需要较低工作量，而82.5%（40个中的33个）需要中等至高工作量才能成功执行。我们识别出五类常见修改需求和13种导致执行失败的挑战，涵盖环境、文档和结构性问题。在可执行的研究产物中，仅35%（40个中的14个）成功复现了原始结果。这些发现凸显了研究产物可用性、可执行性与可复现性之间的显著差距。本研究提出三项可操作的指导原则，以改进研究产物的准备、文档记录和评审工作，从而增强SE研究中开放科学实践的严谨性与可持续性。