Flaky tests, tests that pass or fail nondeterministically without changes to code or environment, pose a serious threat to software reliability. While classical software engineering has developed a rich body of techniques to study flakiness, corresponding evidence for quantum software remains limited. Prior work relies mainly on static analysis or small sets of manually reported incidents, leaving open questions about their prevalence, characteristics, and detectability. This paper presents the first large-scale dynamic characterization of flaky tests in quantum software, focusing on the Qiskit Terra core library. We executed the Qiskit Terra test suite 10,000 times across 23 releases in controlled environments. For each release, we measured test-outcome variability, identified flaky tests, estimated empirical failure probabilities, analyzed recurrence across versions, used Wilson confidence intervals to quantify rerun budgets for reliable detection, and mapped flaky tests to Terra subcomponents. Across 27,026 fully qualified test identifiers, we identified 62 unique flaky tests. Although overall flakiness rates were low (0-0.17%), recurrence was substantial: 52 of 62 flaky tests (83.87%) reappeared in multiple releases, while only 10 tests (16.13%) were confined to a single release. Empirical failure probabilities spanned several orders of magnitude, with a median of $9 \times 10^{-4}$ and 34 tests (54.84%) at or below $10^{-3}$, implying that thousands to tens of thousands of executions may be required for confident detection. These results show that quantum test flakiness is rare but difficult to detect under typical continuous integration budgets. To support future research, we release a public dataset of per-test execution outcomes.

翻译：不稳定性测试（即在不更改代码或环境的情况下，通过或失败结果不确定的测试）对软件可靠性构成严重威胁。尽管经典软件工程已开发出丰富的研究技术来探索不稳定性，但关于量子软件的相应证据仍然有限。先前的工作主要依赖静态分析或少量人工报告案例，其普遍性、特征及可检测性仍存在未解问题。本文首次针对量子软件中的不稳定性测试进行大规模动态表征研究，重点聚焦于Qiskit Terra核心库。我们在受控环境下，对Qiskit Terra测试套件在23个版本中各执行10,000次。针对每个版本，我们测量了测试结果的变异性，识别出不稳定性测试，估计了经验性失败概率，分析了跨版本的复现情况，利用Wilson置信区间量化了可靠检测所需的重复运行预算，并将不稳定性测试映射至Terra子组件。在27,026个完全限定的测试标识符中，我们识别出62个独特的不稳定性测试。尽管整体不稳定性率较低（0-0.17%），但复现性显著：62个不稳定性测试中有52个（83.87%）在多个版本中重复出现，仅10个测试（16.13%）局限于单一版本。经验性失败概率跨越数个数量级，中位数为$9 \times 10^{-4}$，其中34个测试（54.84%）的概率等于或低于$10^{-3}$，这意味着可能需要数千至数万次执行才能实现可靠检测。这些结果表明，量子测试不稳定性虽罕见，但在典型持续集成预算下难以检测。为支持未来研究，我们公开了一个包含每次测试执行结果的数据集。