Large Language Models (LLMs) have demonstrated exceptional proficiency in mathematical reasoning tasks due to their extensive parameter counts and training on vast datasets. Despite these capabilities, deploying LLMs is hindered by their computational demands. Distilling LLM mathematical reasoning into Smaller Language Models (SLMs) has emerged as a solution to this challenge, although these smaller models often suffer from errors in calculation and semantic understanding. Prior work has proposed Program-of-Thought Distillation (PoTD) to avoid calculation error. To further address semantic understanding errors, we propose Key-Point-Driven Mathematical Reasoning Distillation (KPDD). KPDD enhances the reasoning performance of SLMs by breaking down the problem-solving process into three stages: Core Question Extraction, Problem-Solving Information Extraction, and Step-by-Step Solution. This method is further divided into KPDD-CoT, which generates Chain-of-Thought rationales, and KPDD-PoT, which creates Program-of-Thought rationales. The experiment results show that KPDD-CoT significantly improves reasoning abilities, while KPDD-PoT achieves state-of-the-art performance in mathematical reasoning tasks. Our approach effectively mitigates misunderstanding errors, advancing the deployment of efficient and capable SLMs.

翻译：大型语言模型（LLMs）凭借其庞大的参数量和对海量数据集的训练，在数学推理任务中展现出卓越的能力。然而，这些模型的部署受限于其高昂的计算需求。将LLM的数学推理能力蒸馏到更小的语言模型（SLMs）中已成为应对这一挑战的解决方案，尽管这些小型模型常常在计算和语义理解方面存在错误。先前的研究提出了程序思维蒸馏（PoTD）以避免计算错误。为了进一步解决语义理解错误，我们提出了关键点驱动的数学推理蒸馏（KPDD）。KPDD通过将问题求解过程分解为三个阶段：核心问题提取、解题信息提取和分步求解，从而提升SLMs的推理性能。该方法进一步细分为生成思维链推理过程的KPDD-CoT，以及生成程序思维推理过程的KPDD-PoT。实验结果表明，KPDD-CoT显著提升了推理能力，而KPDD-PoT在数学推理任务中实现了最先进的性能。我们的方法有效缓解了理解错误，推动了高效且能力强的小型语言模型的部署。