This work addresses the challenge of democratizing advanced Large Language Models (LLMs) by compressing their mathematical reasoning capabilities into sub-billion parameter Small Language Models (SLMs) without compromising performance. We introduce Equation-of-Thought Distillation (EoTD), a novel technique that encapsulates the reasoning process into equation-based representations to construct an EoTD dataset for fine-tuning SLMs. Additionally, we propose the Mix Thoughts Distillation (MTD) framework to enhance the reasoning performance of SLMs. This involves creating a reasoning dataset with multiple thought processes and using it for fine-tuning. Our experimental findings demonstrate that EoTD significantly boosts the reasoning abilities of SLMs, while MTD enables these models to achieve state-of-the-art reasoning performance.

翻译：本研究旨在解决先进大型语言模型（LLMs）的民主化挑战，通过将其数学推理能力压缩至参数不足十亿的小语言模型（SLMs）中，同时不牺牲性能。我们提出了一种名为“方程思维蒸馏”（EoTD）的新技术，该方法将推理过程封装为基于方程的表征，从而构建用于微调SLMs的EoTD数据集。此外，我们提出了“混合思维蒸馏”（MTD）框架，以增强SLMs的推理性能。该框架通过构建包含多种思维过程的推理数据集，并利用该数据集进行微调。实验结果表明，EoTD显著提升了SLMs的推理能力，而MTD则使这些模型实现了最先进的推理性能。