Non-deterministically behaving (i.e., flaky) tests hamper regression testing as they destroy trust and waste computational and human resources. Eradicating flakiness in test suites is therefore an important goal, but automated debugging tools are needed to support developers when trying to understand the causes of flakiness. A popular example for an automated approach to support regular debugging is spectrum-based fault localization (SFL), a technique that identifies software components that are most likely the causes of failures. While it is possible to also apply SFL for locating likely sources of flakiness in code, unfortunately the flakiness makes SFL both imprecise and non-deterministic. In this paper we introduce SFFL (Spectrum-based Flaky Fault Localization), an extension of traditional coverage-based SFL that exploits our observation that 80% of flaky tests exhibit varying coverage behavior between different runs. By distinguishing between stable and flaky coverage, SFFL is able to locate the sources of flakiness more precisely and keeps the localization itself deterministic. An evaluation on 101 flaky tests taken from 48 open-source Python projects demonstrates that SFFL is effective: Of five prominent SFL formulas, DStar, Ochiai, and Op2 yield the best overall performance. On average, they are able to narrow down the fault's location to 3.5 % of the project's code base, which is 18.7 % better than traditional SFL (for DStar). SFFL's effectiveness, however, depends on the root causes of flakiness: The source of non-order-dependent flaky tests can be located far more precisely than order-dependent faults.

翻译：非确定性行为（即不稳定）的测试会破坏回归测试的可信度并浪费计算和人力资源。因此，消除测试套件中的不稳定性是一个重要目标，但需要自动化调试工具来支持开发人员理解不稳定性的成因。频谱故障定位（SFL）是支持常规调试的自动化方法的一个典型示例，该技术可识别最可能导致故障的软件组件。虽然理论上也可将SFL用于定位代码中可能的不稳定源，但不幸的是，不稳定性导致SFL既不够精确又存在非确定性。本文提出SFFL（基于频谱的不稳定故障定位），这是传统覆盖型SFL的扩展技术，其核心发现是80%的不稳定测试在不同运行间存在变化的覆盖行为。通过区分稳定覆盖与不稳定覆盖，SFFL能够更精确地定位不稳定源头，并保持定位过程本身的确定性。对来自48个开源Python项目的101个不稳定测试进行的评估表明，SFFL效果显著：在五种主流SFL公式中，DStar、Ochiai和Op2表现最佳。平均而言，它们能将故障位置缩小至项目代码库的3.5%，较传统SFL（DStar）提升18.7%。然而，SFFL的有效性取决于不稳定性的根本原因：非顺序依赖型不稳定测试的根源定位比顺序依赖型故障精确得多。