We introduce SuperCLUE-Math6(SC-Math6), a new benchmark dataset to evaluate the mathematical reasoning abilities of Chinese language models. SC-Math6 is designed as an upgraded Chinese version of the GSM8K dataset with enhanced difficulty, diversity, and application scope. It consists of over 2000 mathematical word problems requiring multi-step reasoning and providing natural language solutions. We propose an innovative scheme to quantify the reasoning capability of large models based on performance over problems with different reasoning steps. Experiments on 12 representative Chinese models demonstrate a clear stratification of reasoning levels, with top models like GPT-4 showing superior performance. SC-Math6 fills the gap in Chinese mathematical reasoning benchmarks and provides a comprehensive testbed to advance the intelligence of Chinese language models.

翻译：本文提出SuperCLUE-Math6（SC-Math6），一个用于评估中文语言模型数学推理能力的新型基准数据集。SC-Math6作为GSM8K数据集的中文升级版，在难度、多样性和应用范围上均得到增强。该数据集包含超过2000个需要多步推理的数学应用题，并提供自然语言解答方案。我们提出了一种创新性方案，基于模型在不同推理步骤问题上的表现，量化其推理能力。对12个代表性中文模型的实验表明，推理水平呈现明确分层，其中GPT-4等顶级模型展现出卓越性能。SC-Math6填补了中文数学推理基准的空白，为提升中文语言模型的智能水平提供了全面的测试平台。