With the advent of large language models (LLMs) like GPT-3, a natural question is the extent to which these models can be utilized for source code optimization. This paper presents methodologically stringent case studies applied to well-known open source python libraries pillow and numpy. We find that contemporary LLM ChatGPT-4 (state September and October 2023) is surprisingly adept at optimizing energy and compute efficiency. However, this is only the case in interactive use, with a human expert in the loop. Aware of experimenter bias, we document our qualitative approach in detail, and provide transcript and source code. We start by providing a detailed description of our approach in conversing with the LLM to optimize the _getextrema function in the pillow library, and a quantitative evaluation of the performance improvement. To demonstrate qualitative replicability, we report further attempts on another locus in the pillow library, and one code locus in the numpy library, to demonstrate generalization within and beyond a library. In all attempts, the performance improvement is significant (factor up to 38). We have also not omitted reporting of failed attempts (there were none). We conclude that LLMs are a promising tool for code optimization in open source libraries, but that the human expert in the loop is essential for success. Nonetheless, we were surprised by how few iterations were required to achieve substantial performance improvements that were not obvious to the expert in the loop. We would like bring attention to the qualitative nature of this study, more robust quantitative studies would need to introduce a layer of selecting experts in a representative sample -- we invite the community to collaborate.

翻译：随着 GPT-3 等大型语言模型（LLM）的出现，一个自然的问题是这些模型在多大程度上可用于源代码优化。本文针对著名的开源 Python 库 Pillow 和 NumPy，开展了方法论严谨的案例研究。我们发现，当代 LLM ChatGPT-4（2023 年 9 月和 10 月版本）在优化能源与计算效率方面出人意料地擅长。然而，这仅发生在人机交互场景中，且需人类专家参与循环。为规避实验者偏差，我们详细记录了定性方法，并提供了对话记录与源代码。我们首先详细描述了与 LLM 对话以优化 Pillow 库中 _getextrema 函数的方法，并对性能提升进行了定量评估。为证明定性可复现性，我们进一步报告了针对 Pillow 库中另一处代码段及 NumPy 库中一处代码段的尝试，以展示库内及跨库的泛化能力。所有尝试均实现了显著的性能提升（最高达 38 倍）。我们亦未遗漏失败尝试的报告（实际无失败案例）。我们得出结论：LLM 是开源库代码优化的有前景工具，但人类专家在循环中的参与对成功至关重要。尽管如此，我们惊讶于仅需极少的迭代次数即可实现参与循环专家也无法轻易察觉的实质性性能改进。我们提请关注本研究的定性性质，更稳健的定量研究需引入代表性样本中专家选择的层面——我们诚邀学界同仁合作。