Non-deterministically behaving test cases cause developers to lose trust in their regression test suites and to eventually ignore failures. Detecting flaky tests is therefore a crucial task in maintaining code quality, as it builds the necessary foundation for any form of systematic response to flakiness, such as test quarantining or automated debugging. Previous research has proposed various methods to detect flakiness, but when trying to deploy these in an industrial context, their reliance on instrumentation, test reruns, or language-specific artifacts was inhibitive. In this paper, we therefore investigate the prediction of flaky tests without such requirements on the underlying programming language, CI, build or test execution framework. Instead, we rely only on the most commonly available artifacts, namely the tests' outcomes and durations, as well as basic information about the code evolution to build predictive models capable of detecting flakiness. Furthermore, our approach does not require additional reruns, since it gathers this data from existing test executions. We trained several established classifiers on the suggested features and evaluated their performance on a large-scale industrial software system, from which we collected a data set of 100 flaky and 100 non-flaky test- and code-histories. The best model was able to achieve an F1-score of 95.5% using only 3 features: the tests' flip rates, the number of changes to source files in the last 54 days, as well as the number of changed files in the most recent pull request.

翻译：非确定性行为测试用例导致开发者对其回归测试套件失去信任，并最终忽略测试失败。因此，检测易碎测试是维持代码质量的关键任务，因为它为任何形式的系统性应对易碎性（如测试隔离或自动调试）奠定了必要基础。先前研究提出了多种检测易碎性的方法，但在工业环境中部署时，这些方法对插桩、测试重运行或语言特定产物的依赖构成了阻碍。为此，本文研究无需对底层编程语言、持续集成、构建或测试执行框架提出此类要求的易碎测试预测方法。取而代之，我们仅依赖最常见的可用数据，即测试结果与执行时长，以及代码演化的基本信息，来构建能够检测易碎性的预测模型。此外，由于本方法从现有测试执行中收集数据，因此无需额外重运行测试。我们基于所提出的特征训练了多种经典分类器，并在一个大规模工业软件系统上评估其性能，从中收集了包含100个易碎测试和100个非易碎测试的测试-代码历史数据集。最佳模型仅使用3个特征即可达到95.5%的F1分数：测试的翻转率、过去54天内源文件的变更次数，以及最近一次拉取请求中的变更文件数量。