There has been a recent explosion of research on Large Language Models (LLMs) for software engineering tasks, in particular code generation. However, results from LLMs can be highly unstable; nondeterministically returning very different codes for the same prompt. Non-determinism is a potential menace to scientific conclusion validity. When non-determinism is high, scientific conclusions simply cannot be relied upon unless researchers change their behaviour to control for it in their empirical analyses. This paper conducts an empirical study to demonstrate that non-determinism is, indeed, high, thereby underlining the need for this behavioural change. We choose to study ChatGPT because it is already highly prevalent in the code generation research literature. We report results from a study of 829 code generation problems from three code generation benchmarks (i.e., CodeContests, APPS, and HumanEval). Our results reveal high degrees of non-determinism: the ratio of coding tasks with zero equal test output across different requests is 72.73%, 60.40%, and 65.85% for CodeContests, APPS, and HumanEval, respectively. In addition, we find that setting the temperature to 0 does not guarantee determinism in code generation, although it indeed brings less non-determinism than the default configuration (temperature=1). These results confirm that there is, currently, a significant threat to scientific conclusion validity. In order to put LLM-based research on firmer scientific foundations, researchers need to take into account non-determinism in drawing their conclusions.

翻译：近期涌现大量关于将大语言模型应用于软件工程任务（尤其是代码生成）的研究。然而，LLM的输出可能存在高度不稳定性：对同一提示词会非确定性地返回截然不同的代码。非确定性对科学结论有效性构成潜在威胁。当非确定性显著时，若研究者不在实证分析中加以控制，科学结论将不可信赖。本文通过实证研究证明非确定性确实较高，从而凸显改变研究行为的必要性。我们选择研究ChatGPT，因其在代码生成研究文献中已高度普及。我们报告了来自三个代码生成基准（CodeContests、APPS和HumanEval）共829个代码生成问题的研究结果。结果显示非确定性程度极高：在CodeContests、APPS和HumanEval中，不同请求间测试输出完全一致的编码任务比例分别仅为72.73%、60.40%和65.85%。此外，我们发现将温度参数设为0虽能比默认配置（温度=1）降低非确定性，但无法保证代码生成的确定性。这些结果证实当前科学结论有效性正面临重大威胁。为使基于LLM的研究建立在更坚实的科学基础上，研究者需在得出结论时将非确定性纳入考量。