Non-deterministically behaving test cases cause developers to lose trust in their regression test suites and to eventually ignore failures. Detecting flaky tests is therefore a crucial task in maintaining code quality, as it builds the necessary foundation for any form of systematic response to flakiness, such as test quarantining or automated debugging. Previous research has proposed various methods to detect flakiness, but when trying to deploy these in an industrial context, their reliance on instrumentation, test reruns, or language-specific artifacts was inhibitive. In this paper, we therefore investigate the prediction of flaky tests without such requirements on the underlying programming language, CI, build or test execution framework. Instead, we rely only on the most commonly available artifacts, namely the tests' outcomes and durations, as well as basic information about the code evolution to build predictive models capable of detecting flakiness. Furthermore, our approach does not require additional reruns, since it gathers this data from existing test executions. We trained several established classifiers on the suggested features and evaluated their performance on a large-scale industrial software system, from which we collected a data set of 100 flaky and 100 non-flaky test- and code-histories. The best model was able to achieve an F1-score of 95.5% using only 3 features: the tests' flip rates, the number of changes to source files in the last 54 days, as well as the number of changed files in the most recent pull request.

翻译：非确定性行为的测试用例会导致开发者对其回归测试套件失去信任，并最终忽视测试失败。因此，检测不稳定测试是维护代码质量的关键任务，因为它为任何形式的系统性应对（如测试隔离或自动调试）奠定了必要基础。以往研究提出了多种检测不稳定性的方法，但在工业环境中部署时，这些方法对检测工具、测试重运行或特定语言组件的依赖成为障碍。为此，本文研究无需依赖底层编程语言、CI、构建或测试执行框架的不稳定测试预测方法。我们仅依赖最常用的可用数据（即测试结果和持续时间）以及代码演化的基本信息构建预测模型以检测不稳定性。此外，本方法无需额外重运行，因为它直接从现有测试执行中收集数据。我们在所选特征上训练了多个经典分类器，并在一个大规模工业软件系统中评估其性能，从中收集了100个不稳定和100个稳定测试及其代码演化历史的数据集。最佳模型仅使用三个特征（测试翻转率、最近54天内源文件变更数量，以及最近拉取请求中的变更文件数）即可达到95.5%的F1分数。