With the emergence of Large Language Models (LLMs), there has been a significant improvement in the programming capabilities of models, attracting growing attention from researchers. We propose CodeApex, a bilingual benchmark dataset focusing on the programming comprehension and code generation abilities of LLMs. CodeApex comprises three types of multiple-choice questions: conceptual understanding, commonsense reasoning, and multi-hop reasoning, designed to evaluate LLMs on programming comprehension tasks. Additionally, CodeApex utilizes algorithmic questions and corresponding test cases to assess the code quality generated by LLMs. We evaluate 14 state-of-the-art LLMs, including both general-purpose and specialized models. GPT exhibits the best programming capabilities, achieving approximate accuracies of 50% and 56% on the two tasks, respectively. There is still significant room for improvement in programming tasks. We hope that CodeApex can serve as a reference for evaluating the coding capabilities of LLMs, further promoting their development and growth. Datasets are released at https://github.com/APEXLAB/CodeApex.git. CodeApex submission website is https://apex.sjtu.edu.cn/codeapex/.

翻译：随着大语言模型（LLMs）的出现，模型的编程能力得到了显著提升，引起了研究者越来越多的关注。我们提出了CodeApex，一个专注于评估LLMs编程理解与代码生成能力的双语基准数据集。CodeApex包含三种类型的多选题：概念理解、常识推理和多跳推理，旨在评估LLMs在编程理解任务上的表现。此外，CodeApex利用算法题及相应的测试用例来评估LLMs生成代码的质量。我们评估了14个最先进的LLMs，包括通用模型和专用模型。GPT展现出最佳的编程能力，在两项任务上分别达到了约50%和56%的准确率。编程任务仍有很大的改进空间。我们希望CodeApex能够为评估LLMs的编码能力提供参考，进一步促进其发展和进步。数据集发布在https://github.com/APEXLAB/CodeApex.git。CodeApex提交网站为https://apex.sjtu.edu.cn/codeapex/。