Large language models (LLMs) have demonstrated unparalleled prowess in mimicking human-like text generation and processing. Among the myriad of applications that benefit from LLMs, automated code generation is increasingly promising. The potential to transform natural language prompts into executable code promises a major shift in software development practices and paves the way for significant reductions in manual coding efforts and the likelihood of human-induced errors. This paper reports the results of a study that evaluates the performance of various LLMs, such as Bard, ChatGPT-3.5, ChatGPT-4, and Claude-2, in generating Python for coding problems. We focus on how levels of prompt specificity impact the accuracy, time efficiency, and space efficiency of the generated code. A benchmark of 104 coding problems, each with four types of prompts with varying degrees of tests and specificity, was employed to examine these aspects comprehensively. Our results indicate significant variations in performance across different LLMs and prompt types, and its key contribution is to reveal the ideal prompting strategy for creating accurate Python functions. This study lays the groundwork for further research in LLM capabilities and suggests practical implications for utilizing LLMs in automated code generation tasks and test-driven development.

翻译：大语言模型（LLMs）在模拟人类文本生成与处理方面展现出无与伦比的能力。在众多受益于LLMs的应用中，自动化代码生成前景日益广阔。将自然语言提示转化为可执行代码的潜力，为软件开发实践带来重大变革，有望显著减少人工编码工作量及人为错误概率。本文报告了一项研究结果，该研究评估了Bard、ChatGPT-3.5、ChatGPT-4和Claude-2等多种LLMs在编程问题中生成Python代码的性能表现。我们重点探究提示特异度对生成代码的准确性、时间效率和空间效率的影响。采用包含104个编程问题的基准测试集，每个问题配备四种不同测试程度与特异度的提示类型，以全面考察这些方面。研究结果表明，不同LLMs与提示类型间的性能存在显著差异，其核心贡献在于揭示了创建精准Python函数的最优提示策略。本研究为深入探索LLMs能力奠定了基础，并为利用LLMs执行自动化代码生成任务及测试驱动开发提供了实践启示。