This work addresses the challenge of democratizing advanced Large Language Models (LLMs) by compressing their mathematical reasoning capabilities into sub-billion parameter Small Language Models (SLMs) without compromising performance. We introduce Equation-of-Thought Distillation (EoTD), a novel technique that encapsulates the reasoning process into equation-based representations to construct an EoTD dataset for fine-tuning SLMs. Additionally, we propose the Ensemble Thoughts Distillation (ETD) framework to enhance the reasoning performance of SLMs. This involves creating a reasoning dataset with multiple thought processes, including Chain-of-Thought (CoT), Program-of-Thought (PoT), and Equation-of-Thought (EoT), and using it for fine-tuning. Our experimental findings demonstrate that EoTD significantly boosts the reasoning abilities of SLMs, while ETD enables these models to achieve state-of-the-art reasoning performance.

翻译：本研究针对将先进大型语言模型（LLMs）的数学推理能力无损压缩至参数规模低于十亿的小语言模型（SLMs）这一民主化挑战，提出了一种新颖的方程思维蒸馏（EoTD）技术。该技术将推理过程封装为基于方程的表征形式，构建用于微调SLMs的EoTD数据集。同时，我们提出了集成思维蒸馏（ETD）框架以提升SLMs的推理性能，该框架通过构建包含链式思维（CoT）、程序式思维（PoT）及方程思维（EoT）等多重思维过程的推理数据集进行微调。实验结果表明，EoTD能显著提升SLMs的推理能力，而ETD可使这些模型达到最先进的推理性能。