To develop software with optimal performance, even small performance changes need to be identified. Identifying performance changes is challenging since the performance of software is influenced by non-deterministic factors. Therefore, not every performance change is measurable with reasonable effort. In this work, we discuss which performance changes are measurable at code level with reasonable measurement effort and how to identify them. We present (1) an analysis of the boundaries of measuring performance changes, (2) an approach for determining a configuration for reproducible performance change identification, and (3) an evaluation comparing of how well our approach is able to identify performance changes in the application server Jetty compared with the usage of Jetty's own performance regression benchmarks. Thereby, we find (1) that small performance differences are only measurable by fine-grained measurement workloads, (2) that performance changes caused by the change of one operation can be identified using a unit-test-sized workload definition and a suitable configuration, and (3) that using our approach identifies small performance regressions more efficiently than using Jetty's performance regression benchmarks.

翻译：为开发具有最优性能的软件，即使微小的性能变化也需要被识别。由于软件性能受非确定性因素影响，识别性能变化具有挑战性。因此，并非所有性能变化都能以合理的工作量进行测量。本文探讨了在代码级别上，哪些性能变化能以合理的测量工作量被识别，以及如何识别这些变化。我们提出：(1) 对性能变化测量边界的分析，(2) 一种确定可复现性能变化识别配置的方法，以及(3) 一项评估，比较我们的方法与Jetty自身性能回归基准在应用服务器Jetty中识别性能变化的能力。研究表明：(1) 微小的性能差异仅能通过细粒度的测量工作负载来测量；(2) 由单个操作变更导致的性能变化，可通过单元测试级别的工作负载定义及适当配置进行识别；(3) 相较于使用Jetty的性能回归基准，我们的方法能更高效地识别微小的性能退化。