This work addresses the challenge of democratizing advanced Large Language Models (LLMs) by compressing their mathematical reasoning capabilities into sub-billion parameter Small Language Models (SLMs) without compromising performance. We introduce Equation-of-Thought Distillation (EoTD), a novel technique that encapsulates the reasoning process into equation-based representations to construct an EoTD dataset for fine-tuning SLMs. Additionally, we propose the Ensemble Thoughts Distillation (ETD) framework to enhance the reasoning performance of SLMs. This involves creating a reasoning dataset with multiple thought processes, including Chain-of-Thought (CoT), Program-of-Thought (PoT), and Equation-of-Thought (EoT), and using it for fine-tuning. Our experimental findings demonstrate that EoTD significantly boosts the reasoning abilities of SLMs, while ETD enables these models to achieve state-of-the-art reasoning performance.

翻译：本工作针对将先进大语言模型（LLMs）的数学推理能力压缩至参数量低于十亿的小型语言模型（SLMs）而不损失性能的挑战展开研究。我们提出方程思维蒸馏（EoTD）技术，通过将推理过程封装为基于方程的表示形式，构建用于微调SLMs的EoTD数据集。同时，我们提出集成思维蒸馏（ETD）框架以增强SLMs的推理性能。该框架通过生成包含多种思维过程（包括链式思维CoT、程序思维PoT与方程思维EoT）的推理数据集进行微调。实验结果表明，EoTD显著提升了SLMs的推理能力，而ETD则使这些模型达到了最先进的推理性能水平。