Test flakiness is a significant issue in industry, affecting test efficiency and product quality. While extensive research has examined the impact of flaky tests, many root causes remain unexplored, particularly in the context of dynamic languages such as JavaScript. In this paper, we conduct a systematic evaluation of the impact of environmental factors on test flakiness in JavaScript. We first executed test suites across multiple environmental configurations to determine whether changes in the environment could lead to flaky behavior. We selected three environmental factors to manipulate: the operating system, the Node.js version, and the browser. We identified a total of 65 environmental flaky projects, with 28 related to operating system issues, five to Node.js version compatibility, 16 to a combination of operating system and Node.js issues, and 17 related to browser compatibility. To address environmental flakiness, we developed a lightweight mitigation approach, js-env-sanitizer, that can sanitize environmental-related flaky tests by skipping and reporting them (rather than failing), allowing CI builds to continue/succeed without rerunning entire test suites. The tool achieves high accuracy with minimal performance or configuration overhead, and currently supports three popular JavaScript testing frameworks (Jest, Mocha, and Vitest)
翻译:测试不稳定性是工业界面临的重要问题,它影响测试效率与产品质量。尽管已有大量研究探讨了不稳定测试的影响,但其许多根本原因仍未得到充分探索,特别是在JavaScript等动态语言环境中。本文对JavaScript测试中环境因素对测试不稳定性的影响进行了系统评估。我们首先在多种环境配置下执行测试套件,以确定环境变化是否会导致测试行为不稳定。我们选择了三个环境因素进行操控:操作系统、Node.js版本和浏览器。共识别出65个存在环境不稳定性的项目,其中28个与操作系统问题相关,5个与Node.js版本兼容性相关,16个与操作系统和Node.js问题组合相关,17个与浏览器兼容性相关。为应对环境不稳定性,我们开发了一种轻量级缓解方法js-env-sanitizer,该方法可通过跳过并报告与环境相关的不稳定测试(而非直接判定失败),从而在不重新运行整个测试套件的情况下使CI构建流程得以继续/成功。该工具在保持最低性能或配置开销的同时实现了高准确率,目前支持三种主流的JavaScript测试框架(Jest、Mocha和Vitest)。