New NLP benchmarks are urgently needed to align with the rapid development of large language models (LLMs). We present C-Eval, the first comprehensive Chinese evaluation suite designed to assess advanced knowledge and reasoning abilities of foundation models in a Chinese context. C-Eval comprises multiple-choice questions across four difficulty levels: middle school, high school, college, and professional. The questions span 52 diverse disciplines, ranging from humanities to science and engineering. C-Eval is accompanied by C-Eval Hard, a subset of very challenging subjects in C-Eval that requires advanced reasoning abilities to solve. We conduct a comprehensive evaluation of the most advanced LLMs on C-Eval, including both English- and Chinese-oriented models. Results indicate that only GPT-4 could achieve an average accuracy of over 60%, suggesting that there is still significant room for improvement for current LLMs. We anticipate C-Eval will help analyze important strengths and shortcomings of foundation models, and foster their development and growth for Chinese users.

翻译：为紧跟大语言模型的快速发展，亟需构建新的自然语言处理基准。我们提出C-Eval，这是首个专为评估基础模型在中文语境下的高级知识与推理能力而设计的综合性中文评测套件。C-Eval包含涵盖四个难度层次（初中、高中、大学与职业）的选择题，涉及从人文科学到理工科的52个多元学科领域。与C-Eval配套的还有C-Eval Hard子集，该子集聚焦C-Eval中需要高级推理能力的极富挑战性科目。我们对当前最先进的大语言模型（包括面向英文与中文的模型）在C-Eval上进行了全面评估。结果表明，仅GPT-4的平均准确率超过60%，说明现有大语言模型仍有显著提升空间。我们期望C-Eval能助力分析基础模型的重要优势与不足，并促进其为中文用户的发展与进化。