Flaky tests obstruct software development, and studying and proposing mitigations against them has therefore become an important focus of software engineering research. To conduct sound investigations on test flakiness, it is crucial to have large, diverse, and unbiased datasets of flaky tests. A common method to build such datasets is by rerunning the test suites of selected projects multiple times and checking for tests that produce different outcomes. While using this technique on a single project is mostly straightforward, applying it to a large and diverse set of projects raises several implementation challenges such as (1) isolating the test executions, (2) supporting multiple build mechanisms, (3) achieving feasible run times on large datasets, and (4) analyzing and presenting the test outcomes. To address these challenges we introduce FlaPy, a framework for researchers to mine flaky tests in a given or automatically sampled set of Python projects by rerunning their test suites. FlaPy isolates the test executions using containerization and fresh execution environments to simulate real-world CI conditions and to achieve accurate results. By supporting multiple dependency installation strategies, it promotes diversity among the studied projects. FlaPy supports parallelizing the test executions using SLURM, making it feasible to scan thousands of projects for test flakiness. Finally, FlaPy analyzes the test outcomes to determine which tests are flaky and depicts the results in a concise table. A demo video of FlaPy is available at https://youtu.be/ejy-be-FvDY

翻译：不稳定测试会阻碍软件开发，因此研究不稳定测试并提出缓解方法已成为软件工程研究的重要焦点。为了对测试不稳定性进行可靠的研究，获取大规模、多样化且无偏见的不稳定测试数据集至关重要。构建此类数据集的常见方法是通过多次重新运行选定项目的测试套件，并检查产生不同结果的测试。虽然将此技术应用于单个项目大多较为直接，但将其应用于大规模且多样的项目集合时会带来若干实现挑战，例如：(1)隔离测试执行；(2)支持多种构建机制；(3)在大型数据集上实现可行的运行时间；(4)分析并呈现测试结果。为应对这些挑战，我们提出了FlaPy——一个供研究人员在给定或自动采样的Python项目集合中通过重新运行其测试套件来挖掘不稳定测试的框架。FlaPy利用容器化和全新执行环境隔离测试执行，以模拟真实世界的持续集成条件并获取准确结果。通过支持多种依赖安装策略，它促进了所研究项目的多样性。FlaPy支持使用SLURM并行化测试执行，从而可扫描数千个项目以检测测试不稳定性。最后，FlaPy分析测试结果以判定哪些测试为不稳定测试，并以简洁表格形式呈现结果。FlaPy的演示视频见https://youtu.be/ejy-be-FvDY