Test flakiness is a problem that affects testing and processes that rely on it. Several factors cause or influence the flakiness of test outcomes. Test execution order, randomness and concurrency are some of the more common and well-studied causes. Some studies mention code instrumentation as a factor that causes or affects test flakiness. However, evidence for this issue is scarce. In this study, we attempt to systematically collect evidence for the effects of instrumentation on test flakiness. We experiment with common types of instrumentation for Java programs - namely, application performance monitoring, coverage and profiling instrumentation. We then study the effects of instrumentation on a set of nine programs obtained from an existing dataset used to study test flakiness, consisting of popular GitHub projects written in Java. We observe cases where real-world instrumentation causes flakiness in a program. However, this effect is rare. We also discuss a related issue - how instrumentation may interfere with flakiness detection and prevention.

翻译：测试不稳定性是影响测试及其依赖流程的问题。多种因素导致或影响测试结果的波动性。测试执行顺序、随机性和并发性是较为常见且研究充分的原因。部分研究提及代码插桩是导致或影响测试不稳定性的因素之一，但相关证据较为匮乏。本研究旨在系统性地收集插桩对测试不稳定性影响的证据。我们针对Java程序常见的插桩类型——即应用性能监控、覆盖率与性能剖析插桩——开展实验，并基于现有测试不稳定性研究数据集中的九个程序（均为GitHub上流行的Java项目）研究插桩的影响。实验观察到真实场景下插桩确实会引发程序的不稳定性，但此类情况较为罕见。同时，本文探讨了插桩可能干扰测试不稳定性检测与预防的相关问题。