Software testing plays a crucial role in the contribution process of open-source projects. For example, contributions introducing new features are expected to include tests, and contributions with tests are more likely to be accepted. Although most real-world projects require contributors to write tests, the specific testing practices communicated to contributors remain unclear. In this paper, we present an empirical study to understand better how software testing is approached in contribution guidelines. We analyze the guidelines of 200 Python and JavaScript open-source software projects. We find that 78\% of the projects include some form of test documentation for contributors. Test documentation is located in multiple sources, including \texttt{CONTRIBUTING} files (58\%), external documentation (24\%), and \texttt{README} files (8\%). Furthermore, test documentation commonly explains how to run tests (83.5\%), but less often provides guidance on how to write tests (37\%). It frequently covers unit tests (71\%), but rarely addresses integration (20.5\%) and end-to-end tests (15.5\%). Other key testing aspects are also less frequently discussed: test coverage (25.5\%) and mocking (9.5\%). We conclude by discussing implications and future research.
翻译:软件测试在开源项目的贡献流程中扮演着关键角色。例如,引入新功能的贡献通常需要包含测试,而附带测试的贡献更有可能被接受。尽管大多数实际项目都要求贡献者编写测试,但向贡献者传达的具体测试实践仍不明确。本文通过一项实证研究,旨在深入理解贡献指南中如何阐述软件测试方法。我们分析了200个Python与JavaScript开源软件项目的指南。研究发现,78%的项目为贡献者提供了某种形式的测试文档。测试文档分布于多个来源,包括\texttt{CONTRIBUTING}文件(58%)、外部文档(24%)以及\texttt{README}文件(8%)。此外,测试文档通常说明如何运行测试(83.5%),但较少提供编写测试的指导(37%)。文档内容频繁涉及单元测试(71%),却较少涵盖集成测试(20.5%)与端到端测试(15.5%)。其他关键测试方面亦较少被讨论:测试覆盖率(25.5%)与模拟测试(9.5%)。最后,本文讨论了相关启示与未来研究方向。