The transformative influence of Large Language Models (LLMs) is profoundly reshaping the Artificial Intelligence (AI) technology domain. Notably, ChatGPT distinguishes itself within these models, demonstrating remarkable performance in multi-turn conversations and exhibiting code proficiency across an array of languages. In this paper, we carry out a comprehensive evaluation of ChatGPT's coding capabilities based on what is to date the largest catalog of coding challenges. Our focus is on the python programming language and problems centered on data structures and algorithms, two topics at the very foundations of Computer Science. We evaluate ChatGPT for its ability to generate correct solutions to the problems fed to it, its code quality, and nature of run-time errors thrown by its code. Where ChatGPT code successfully executes, but fails to solve the problem at hand, we look into patterns in the test cases passed in order to gain some insights into how wrong ChatGPT code is in these kinds of situations. To infer whether ChatGPT might have directly memorized some of the data that was used to train it, we methodically design an experiment to investigate this phenomena. Making comparisons with human performance whenever feasible, we investigate all the above questions from the context of both its underlying learning models (GPT-3.5 and GPT-4), on a vast array sub-topics within the main topics, and on problems having varying degrees of difficulty.

翻译：大型语言模型（LLMs）的变革性影响正深刻重塑人工智能（AI）技术领域。值得注意的是，ChatGPT在这些模型中脱颖而出，在多轮对话中展现出卓越性能，并在多种语言中表现出编程熟练度。本文基于迄今为止规模最大的编程挑战数据集，对ChatGPT的编码能力进行了全面评估。我们聚焦于Python编程语言以及数据结构和算法问题——这两大计算机科学基石。我们评估了ChatGPT在生成问题正确解决方案方面的能力、代码质量以及其代码产生的运行时错误性质。当ChatGPT代码成功执行但未能解决当前问题时，我们分析其通过的测试用例模式，以深入了解这些情况下ChatGPT代码的错误程度。为推断ChatGPT是否可能直接记忆了部分训练数据，我们系统地设计了一项实验来探究这一现象。在可行的情况下，我们将其与人类表现进行对比，从底层学习模型（GPT-3.5和GPT-4）的视角出发，针对主要主题下的广泛子主题以及不同难度级别的问题，研究了上述所有问题。