The popularity of open-source software (OSS) projects has grown significantly over the last few years with more organizations relying on them. As these projects become larger, the need for higher quality also increases. DevOps practices have been shown to improve quality and performance. The DORA benchmarking reports provide useful information to compare DevOps practices performance between organizations, but they focus on continuous deployment and delivery to production, while OSS projects focus on the continuous release of code and its impact on third parties. The DORA reports mention the increasing presence of OSS projects as they are widely used in the industry, but they have never been used to measure OSS projects performance levels. This study reveals that the DORA benchmark cannot be applied to OSS projects and proposes benchmarking metrics for OSS projects, being the first one that adapts the DORA metrics and applies them in OSS projects. The metrics proposed in this study for benchmarking OSS projects include Release Frequency and Lead Time For Released Changes to measure throughput, and Time To Repair Code and Bug Issues Rate to assess stability. In contrast to the DORA reports, where data is collected through manual surveys, in our proposal, data is collected automatically by a tool we developed that retrieves information from public GitHub repositories. This reduces the risk of survey-based data collection. Our study also shows the benchmark feasibility by applying it to four popular OSS projects: Angular, Kubernetes, Tensorflow, and VS Code. In addition, we proposed challenges that address the topics and future works to expand the knowledge and findings of this study. Overall, the findings of the study can help to improve future research on OSS projects and provide a better understanding and challenges of the role of DevOps practices in OSS projects.

翻译：开源软件（OSS）项目的普及程度在过去几年中显著增长，越来越多的组织依赖这些项目。随着这些项目规模不断扩大，对其质量的要求也随之提高。DevOps实践已被证明能够有效提升软件质量与性能。DORA基准测试报告提供了用于比较组织间DevOps实践绩效的有用信息，但其侧重于持续部署与持续交付到生产环境，而OSS项目则关注代码的持续发布及其对第三方的影响。DORA报告虽提及OSS项目在行业中的广泛应用和日益增长的存在感，但从未被用于衡量OSS项目的绩效水平。本研究表明DORA基准无法直接应用于OSS项目，并首次提出适应于OSS项目的基准测试指标，将DORA指标调整后应用于OSS项目。本研究提出的OSS项目基准测试指标包括：用于衡量吞吐量的发布频率（Release Frequency）与已发布变更的交付周期（Lead Time For Released Changes），以及用于评估稳定性的代码修复时间（Time To Repair Code）和缺陷问题率（Bug Issues Rate）。与DORA报告通过人工调查收集数据的方式不同，我们的方案通过自主研发的工具自动从GitHub公共仓库中检索数据，从而降低了基于调查的数据收集风险。本研究还通过将基准测试应用于Angular、Kubernetes、Tensorflow和VS Code四个主流OSS项目，验证了该基准的可行性。此外，我们提出了相关主题挑战与未来研究方向，以扩展本研究的认知与成果。总体而言，本研究结果有助于改进未来OSS项目研究，并为理解DevOps实践在OSS项目中的作用及其挑战提供更深入的见解。