This work addresses the challenge of democratizing advanced Large Language Models (LLMs) by compressing their mathematical reasoning capabilities into sub-billion parameter Small Language Models (SLMs) without compromising performance. We introduce Equation-of-Thought Distillation (EoTD), a novel technique that encapsulates the reasoning process into equation-based representations to construct an EoTD dataset for fine-tuning SLMs. Additionally, we propose the Ensemble Thoughts Distillation (ETD) framework to enhance the reasoning performance of SLMs. This involves creating a reasoning dataset with multiple thought processes, including Chain-of-Thought (CoT), Program-of-Thought (PoT), and Equation-of-Thought (EoT), and using it for fine-tuning. Our experimental findings demonstrate that EoTD significantly boosts the reasoning abilities of SLMs, while ETD enables these models to achieve state-of-the-art reasoning performance.

翻译：本研究聚焦于将先进大型语言模型（LLMs）的数学推理能力无损压缩至参数量低于十亿的小型语言模型（SLMs），旨在推动高级模型的民主化进程。我们提出了"方程思维蒸馏"（EoTD）这一创新技术，通过将推理过程封装为基于方程的表示形式，构建用于微调SLMs的EoTD数据集。同时，我们设计了"集成思维蒸馏"（ETD）框架来增强SLMs的推理性能，该框架通过构建包含链式思维（CoT）、编程思维（PoT）和方程思维（EoT）等多种思维过程的数据集进行微调。实验结果表明，EoTD能显著提升SLMs的推理能力，而ETD则使这些模型达到最先进的推理性能。