This paper presents a comprehensive evaluation of the code generation capabilities of ChatGPT, a prominent large language model, compared to human programmers. A novel dataset of 131 code-generation prompts across 5 categories was curated to enable robust analysis. Code solutions were generated by both ChatGPT and humans for all prompts, resulting in 262 code samples. A meticulous manual assessment methodology prioritized evaluating correctness, comprehensibility, and security using 14 established code quality metrics. The key findings reveal ChatGPT's strengths in crafting concise, efficient code with advanced constructs, showcasing strengths in data analysis tasks (93.1% accuracy) but limitations in visual-graphical challenges. Comparative analysis with human code highlights ChatGPT's inclination towards modular design and superior error handling. Additionally, machine learning models effectively distinguished ChatGPT from human code with up to 88% accuracy, suggesting detectable coding style disparities. By providing profound insights into ChatGPT's code generation capabilities and limitations through quantitative metrics and qualitative analysis, this study makes valuable contributions toward advancing AI-based programming assistants. The curated dataset and methodology offer a robust foundation for future research in this nascent domain. All data and codes are available on https://github.com/DSAatUSU/ChatGPT-promises-and-pitfalls.

翻译：本文对大型语言模型ChatGPT与人类程序员的代码生成能力进行了全面评估。我们构建了一个包含5个类别、131个代码生成提示的新颖数据集，以实现稳健分析。针对所有提示，ChatGPT与人类分别生成了代码解决方案，共获得262个代码样本。通过严格的逐条人工评估方法，采用14项既定代码质量指标，重点评估了代码的正确性、可理解性和安全性。关键发现揭示了ChatGPT在构建简洁高效代码及运用高级构造方面的优势，在数据分析任务中表现突出（准确率93.1%），但在可视化图形任务中存在局限。与人类代码的对比分析表明，ChatGPT更倾向于模块化设计并具备更优的错误处理能力。此外，机器学习模型对ChatGPT与人类代码的区分准确率高达88%，表明两者存在可检测的编码风格差异。本研究通过定量指标与定性分析深入揭示了ChatGPT的代码生成能力与局限性，为推进基于人工智能的编程助手作出了重要贡献。所构建的数据集与评估方法为该新兴领域的未来研究奠定了坚实基础。所有数据与代码均可在https://github.com/DSAatUSU/ChatGPT-promises-and-pitfalls获取。