Mathematical understanding and reasoning are crucial tasks for assessing the capabilities of artificial intelligence (AI). However, existing benchmarks either require just a few steps of reasoning, or only contain a small amount of data in one specific topic, making it hard to analyse AI's behaviour with reference to different problems within a specific topic in detail. In this work, we propose Conic10K, a challenging math problem dataset on conic sections in Chinese senior high school education. Our dataset contains various problems with different reasoning depths, while only the knowledge from conic sections is required. Since the dataset only involves a narrow range of knowledge, it is easy to separately analyse the knowledge a model possesses and the reasoning ability it has. For each problem, we provide a high-quality formal representation, the reasoning steps, and the final solution. Experiments show that existing large language models, including GPT-4, exhibit weak performance on complex reasoning. We hope that our findings could inspire more advanced techniques for precise natural language understanding and reasoning. Our dataset and codes are available at https://github.com/whyNLP/Conic10K.

翻译：数学理解与推理是评估人工智能（AI）能力的关键任务。然而，现有基准测试要么仅需少量推理步骤，要么仅包含特定主题下的小规模数据，难以针对某一主题内的不同问题详细分析AI的行为。为此，我们提出Conic10K，这是一个面向中国高中数学教育的圆锥曲线专题挑战性数学问题数据集。该数据集包含具有不同推理深度的多样化问题，但仅需圆锥曲线相关知识。由于数据集仅涉及狭窄的知识范围，可以便捷地独立分析模型所具备的知识与推理能力。针对每个问题，我们提供高质量的形式化表示、推理步骤及最终解答。实验表明，包括GPT-4在内的现有大型语言模型在复杂推理任务上表现欠佳。我们期待这一发现能够启发更先进的精确自然语言理解与推理技术。我们的数据集与代码已开源在https://github.com/whyNLP/Conic10K。