Evaluating the general abilities of foundation models to tackle human-level tasks is a vital aspect of their development and application in the pursuit of Artificial General Intelligence (AGI). Traditional benchmarks, which rely on artificial datasets, may not accurately represent human-level capabilities. In this paper, we introduce AGIEval, a novel benchmark specifically designed to assess foundation model in the context of human-centric standardized exams, such as college entrance exams, law school admission tests, math competitions, and lawyer qualification tests. We evaluate several state-of-the-art foundation models, including GPT-4, ChatGPT, and Text-Davinci-003, using this benchmark. Impressively, GPT-4 surpasses average human performance on SAT, LSAT, and math competitions, attaining a 95% accuracy rate on the SAT Math test and a 92.5% accuracy on the English test of the Chinese national college entrance exam. This demonstrates the extraordinary performance of contemporary foundation models. In contrast, we also find that GPT-4 is less proficient in tasks that require complex reasoning or specific domain knowledge. Our comprehensive analyses of model capabilities (understanding, knowledge, reasoning, and calculation) reveal these models' strengths and limitations, providing valuable insights into future directions for enhancing their general capabilities. By concentrating on tasks pertinent to human cognition and decision-making, our benchmark delivers a more meaningful and robust evaluation of foundation models' performance in real-world scenarios. The data, code, and all model outputs are released in https://github.com/ruixiangcui/AGIEval.

翻译：评估基础模型处理人类层级任务的一般能力，是其在通往通用人工智能（AGI）发展与应用中的关键环节。传统依赖人工数据集的基准测试难以准确反映人类层级能力。本文提出AGIEval——一个专门针对人类中心标准化考试（如高考、法学院入学考试、数学竞赛及律师资格考试）设计的新型评估基准。我们利用该基准对GPT-4、ChatGPT和Text-Davinci-003等前沿基础模型进行了评估。令人瞩目的是，GPT-4在SAT、LSAT及数学竞赛中超越人类平均水平，SAT数学测试准确率达95%，中国高考英语测试准确率达92.5%，充分展现了当代基础模型的卓越性能。同时我们发现，GPT-4在需要复杂推理或特定领域知识的任务中表现欠佳。通过对模型能力（理解、知识、推理与计算）的全面分析，揭示了这些模型的优势与局限，为提升其通用能力提供了重要启示。由于本基准聚焦于人类认知与决策相关任务，能更有效、更可靠地评估基础模型在真实场景中的表现。相关数据、代码及所有模型输出已发布于https://github.com/ruixiangcui/AGIEval。