In the burgeoning field of artificial intelligence (AI), understanding the capabilities and limitations of programming-oriented models is crucial. This paper presents a novel evaluation of the programming proficiency of Generative Pretrained Transformer (GPT) models, specifically GPT-3.5 and GPT-4, against coding problems of varying difficulty levels drawn from Codewars. The experiments reveal a distinct boundary at the 3kyu level, beyond which these GPT models struggle to provide solutions. These findings led to the proposal of a measure for coding problem complexity that incorporates both problem difficulty and the time required for solution. The research emphasizes the need for validation and creative thinking capabilities in AI models to better emulate human problem-solving techniques. Future work aims to refine this proposed complexity measure, enhance AI models with these suggested capabilities, and develop an objective measure for programming problem difficulty. The results of this research offer invaluable insights for improving AI programming capabilities and advancing the frontier of AI problem-solving abilities.

翻译：在人工智能（AI）蓬勃发展的领域中，理解面向编程模型的能力与局限性至关重要。本文针对生成式预训练Transformer（GPT）模型（具体为GPT-3.5和GPT-4）的编程熟练度，提出了一种新颖的评估方法，所依据的是来自Codewars的、涵盖不同难度级别的编程问题。实验揭示了一个清晰的分界点，即3kyu级别，超出此级别后，这些GPT模型便难以提供解决方案。这些发现促使我们提出一种编码问题复杂度的度量方法，该方法同时考虑了问题难度与解题所需时间。本研究强调了AI模型在验证与创造性思维能力方面的需求，以使其更好地模拟人类问题解决技巧。未来工作旨在完善这一提出的复杂度度量方法，为AI模型赋予这些建议的能力，并开发一种针对编程问题难度的客观衡量标准。本研究结果为提升AI编程能力、推进AI问题解决能力的前沿提供了宝贵的见解。