New NLP benchmarks are urgently needed to align with the rapid development of large language models (LLMs). We present C-Eval, the first comprehensive Chinese evaluation suite designed to assess advanced knowledge and reasoning abilities of foundation models in a Chinese context. C-Eval comprises multiple-choice questions across four difficulty levels: middle school, high school, college, and professional. The questions span 52 diverse disciplines, ranging from humanities to science and engineering. C-Eval is accompanied by C-Eval Hard, a subset of very challenging subjects in C-Eval that requires advanced reasoning abilities to solve. We conduct a comprehensive evaluation of the most advanced LLMs on C-Eval, including both English- and Chinese-oriented models. Results indicate that only GPT-4 could achieve an average accuracy of over 60%, suggesting that there is still significant room for improvement for current LLMs. We anticipate C-Eval will help analyze important strengths and shortcomings of foundation models, and foster their development and growth for Chinese users.

翻译：新的自然语言处理基准亟需与大型语言模型（LLMs）的快速发展同步。我们提出C-Eval，首个旨在中文语境下评估基础模型高级知识与推理能力的综合性中文评估套件。C-Eval包含涵盖四个难度等级的多项选择题：初中、高中、大学及专业级别。题目横跨52个不同学科，涵盖从人文学科到科学与工程领域。C-Eval配套推出C-Eval Hard，即C-Eval中需要高级推理能力才能解决的极具挑战性学科子集。我们对最先进的大型语言模型（包括面向英语与中文的模型）在C-Eval上进行了全面评估。结果显示，仅GPT-4的平均准确率超过60%，表明当前LLMs仍有显著提升空间。我们预期C-Eval将有助于分析基础模型的重要优势与不足，并促进其为中文用户的发展与成长。