Automated regression testing is a cornerstone of modern software development, often contributing directly to code review and Continuous Integration (CI). Yet some tests suffer from flakiness, where their outcomes vary non-deterministically. Flakiness erodes developer trust in test results, wastes computational resources, and undermines CI reliability. While prior research has examined test flakiness within individual projects, its broader ecosystem-wide impact remains largely unexplored. In this paper, we present an empirical study of test flakiness in the OpenStack ecosystem, which focuses on (1) cross-project flakiness, where flaky tests impact multiple projects, and (2) inconsistent flakiness, where a test exhibits flakiness in some projects but remains stable in others. By analyzing 649 OpenStack projects, we identify 1,535 cross-project flaky tests and 1,105 inconsistently flaky tests. We find that cross-project flakiness affects 55% of OpenStack projects and significantly increases both review time and computational costs. Surprisingly, 70% of unit tests exhibit cross-project flakiness, challenging the assumption that unit tests are inherently insulated from issues that span modules like integration and system-level tests. Through qualitative analysis, we observe that race conditions in CI, inconsistent build configurations, and dependency mismatches are the primary causes of inconsistent flakiness. These findings underline the need for better coordination across complex ecosystems, standardized CI configurations, and improved test isolation strategies.

翻译：自动化回归测试是现代软件开发的基石，通常直接促进代码审查和持续集成（CI）。然而，某些测试存在不稳定性，其结果会非确定性地变化。测试不稳定性削弱了开发者对测试结果的信任，浪费计算资源，并损害CI的可靠性。尽管先前的研究已考察了单个项目内部的测试不稳定性，但其在更广泛的生态系统中的影响仍未充分探索。在本文中，我们对OpenStack生态系统中的测试不稳定性进行了实证研究，重点关注：(1) 跨项目不稳定性，即不稳定的测试影响多个项目；(2) 不一致不稳定性，即某个测试在某些项目中表现出不稳定性，而在其他项目中保持稳定。通过分析649个OpenStack项目，我们识别出1,535个跨项目不稳定测试和1,105个不一致不稳定测试。我们发现，跨项目不稳定性影响了55%的OpenStack项目，并显著增加了审查时间和计算成本。令人惊讶的是，70%的单元测试表现出跨项目不稳定性，这挑战了“单元测试天生与集成测试和系统级测试等跨模块问题绝缘”的假设。通过定性分析，我们观察到CI中的竞争条件、不一致的构建配置以及依赖关系不匹配是不一致不稳定性的主要原因。这些发现强调了在复杂生态系统中加强协调、标准化CI配置以及改进测试隔离策略的必要性。