With the advent of large language models (LLMs) like GPT-3, a natural question is the extent to which these models can be utilized for source code optimization. This paper presents methodologically stringent case studies applied to well-known open source python libraries pillow and numpy. We find that contemporary LLM ChatGPT-4 (state September and October 2023) is surprisingly adept at optimizing energy and compute efficiency. However, this is only the case in interactive use, with a human expert in the loop. Aware of experimenter bias, we document our qualitative approach in detail, and provide transcript and source code. We start by providing a detailed description of our approach in conversing with the LLM to optimize the _getextrema function in the pillow library, and a quantitative evaluation of the performance improvement. To demonstrate qualitative replicability, we report further attempts on another locus in the pillow library, and one code locus in the numpy library, to demonstrate generalization within and beyond a library. In all attempts, the performance improvement is significant (factor up to 38). We have also not omitted reporting of failed attempts (there were none). We conclude that LLMs are a promising tool for code optimization in open source libraries, but that the human expert in the loop is essential for success. Nonetheless, we were surprised by how few iterations were required to achieve substantial performance improvements that were not obvious to the expert in the loop. We would like bring attention to the qualitative nature of this study, more robust quantitative studies would need to introduce a layer of selecting experts in a representative sample -- we invite the community to collaborate.

翻译：随着GPT-3等大型语言模型的出现，一个自然的问题便是这些模型在源代码优化中的应用程度。本文针对知名开源Python库Pillow和NumPy开展了方法学严谨的案例研究。我们发现，当代LLM ChatGPT-4（2023年9月至10月状态）在优化能源和计算效率方面表现出惊人的能力，但这仅发生在人类专家参与交互式使用时。为克服实验者偏差，我们详细记录了定性研究方法，并提供对话记录与源代码。我们首先详细描述与LLM对话优化Pillow库内_getextrema函数的流程，并对性能提升进行定量评估。为验证定性结果的可复现性，我们报告了针对Pillow库另一代码段及NumPy库内一个代码段的进一步实验，以证明跨库及库内泛化能力。所有尝试均实现显著性能提升（最高达38倍）。我们亦未遗漏任何失败案例的汇报（实际无失败案例）。研究结论表明，LLM是开源库代码优化的有力工具，但人类专家的参与对成功至关重要。不过，我们惊讶于仅需极少量迭代即可获得人类专家也难以预见的显著性能提升。需指出本研究的定性本质，更严谨的定量研究需引入代表性专家样本的筛选机制——我们诚邀学界同仁共同推进。