Catching and attributing code change-induced performance regressions in production is hard; predicting them beforehand, even harder. A primer on automatically learning to predict performance regressions in software, this article gives an account of the experiences we gained when researching and deploying an ML-based regression prediction pipeline at Meta. In this paper, we report on a comparative study with four ML models of increasing complexity, from (1) code-opaque, over (2) Bag of Words, (3) off-the-shelve Transformer-based, to (4) a bespoke Transformer-based model, coined SuperPerforator. Our investigation shows the inherent difficulty of the performance prediction problem, which is characterized by a large imbalance of benign onto regressing changes. Our results also call into question the general applicability of Transformer-based architectures for performance prediction: an off-the-shelve CodeBERT-based approach had surprisingly poor performance; our highly customized SuperPerforator architecture initially achieved prediction performance that was just on par with simpler Bag of Words models, and only outperformed them for down-stream use cases. This ability of SuperPerforator to transfer to an application with few learning examples afforded an opportunity to deploy it in practice at Meta: it can act as a pre-filter to sort out changes that are unlikely to introduce a regression, truncating the space of changes to search a regression in by up to 43%, a 45x improvement over a random baseline. To gain further insight into SuperPerforator, we explored it via a series of experiments computing counterfactual explanations. These highlight which parts of a code change the model deems important, thereby validating the learned black-box model.

翻译：捕捉并归因生产环境中由代码变更引发的性能回归十分困难；而提前预测则更为艰难。作为一篇关于自动学习预测软件性能回归的入门指南，本文阐述了我们在Meta研究并部署基于机器学习的回归预测流水线时积累的经验。本文报告了一项比较研究，涉及四种复杂度递增的机器学习模型：(1) 代码无关模型，(2) 词袋模型，(3) 基于现成Transformer的模型，以及(4) 定制化Transformer模型（称为SuperPerforator）。我们的研究表明，性能预测问题固有难度在于良性变更与回归变更之间存在严重不平衡。研究结果还质疑了基于Transformer架构在性能预测中的通用适用性：基于现成CodeBERT的方法表现差强人意；高度定制的SuperPerforator架构初始预测性能仅与简单词袋模型持平，且仅在下游应用中优于后者。SuperPerforator能够通过少量学习样本迁移至实际应用场景，这使其得以在Meta实践中部署：它可作为预过滤器，筛选出不太可能引入回归的变更，将搜索回归的变更空间缩减达43%，相比随机基线提升45倍。为深入理解SuperPerforator，我们通过一系列计算反事实解释的实验进行探索。这些实验揭示了模型认为代码变更中的哪些部分至关重要，从而验证了所学习的黑箱模型。