We systematically study the quality of 4,066 ChatGPT-generated code implemented in two popular programming languages, i.e., Java and Python, for 2,033 programming tasks. The goal of this work is three folds. First, we analyze the correctness of ChatGPT on code generation tasks and uncover the factors that influence its effectiveness, including task difficulty, programming language, time that tasks are introduced, and program size. Second, we identify and characterize potential issues with the quality of ChatGPT-generated code. Last, we provide insights into how these issues can be mitigated. Experiments highlight that out of 4,066 programs generated by ChatGPT, 2,756 programs are deemed correct, 1,082 programs provide wrong outputs, and 177 programs contain compilation or runtime errors. Additionally, we further analyze other characteristics of the generated code through static analysis tools, such as code style and maintainability, and find that 1,930 ChatGPT-generated code snippets suffer from maintainability issues. Subsequently, we investigate ChatGPT's self-repairing ability and its interaction with static analysis tools to fix the errors uncovered in the previous step. Experiments suggest that ChatGPT can partially address these challenges, improving code quality by more than 20%, but there are still limitations and opportunities for improvement. Overall, our study provides valuable insights into the current limitations of ChatGPT and offers a roadmap for future research and development efforts to enhance the code generation capabilities of AI models like ChatGPT.

翻译：我们系统性地研究了由 ChatGPT 生成的 4,066 个代码片段的质量，这些代码涉及两种主流编程语言（Java 和 Python），覆盖 2,033 个编程任务。本研究的目标有三：首先，分析 ChatGPT 在代码生成任务中的正确性，并揭示影响其效能的因素，包括任务难度、编程语言、任务引入时间及程序规模；其次，识别并表征 ChatGPT 生成代码中潜在的质量问题；最后，探讨如何缓解这些问题的见解。实验表明，在 ChatGPT 生成的 4,066 个程序中，2,756 个程序正确，1,082 个程序输出错误，177 个程序存在编译或运行时错误。此外，我们通过静态分析工具（如代码风格与可维护性）进一步分析生成代码的其他特性，发现 1,930 个 ChatGPT 生成代码片段存在可维护性问题。随后，我们探究 ChatGPT 的自修复能力及其与静态分析工具的交互，以修正前一阶段发现的错误。实验表明，ChatGPT 能部分应对这些挑战，将代码质量提升超过 20%，但仍存在局限性与改进空间。总体而言，本研究揭示了 ChatGPT 当前局限性，并为未来人工智能模型（如 ChatGPT）代码生成能力的研发工作提供了路线图。