Small-scale models offer various computational advantages, and yet to which extent size is critical for problem-solving abilities remains an open question. Specifically for solving grade school math, the smallest model size so far required to break the 80\% barrier on the GSM8K benchmark remains to be 34B. Our work studies how high-quality datasets may be the key for small language models to acquire mathematical reasoning. We introduce \texttt{TinyGSM}, a synthetic dataset of 12.3M grade school math problems paired with Python solutions, generated fully by GPT-3.5. After finetuning on \texttt{TinyGSM}, we find that a duo of a 1.3B generation model and a 1.3B verifier model can achieve 81.5\% accuracy, outperforming existing models that are orders of magnitude larger. This also rivals the performance of the GPT-3.5 ``teacher'' model (77.4\%), from which our model's training data is generated. Our approach is simple and has two key components: 1) the high-quality dataset \texttt{TinyGSM}, 2) the use of a verifier, which selects the final outputs from multiple candidate generations.

翻译：小规模模型在各种计算任务上具有优势，但模型规模对解决数学问题的能力起多大作用仍是悬而未决的问题。在小学数学领域，此前突破GSM8k基准80%准确率所需的最小模型规模为34B参数。本研究聚焦高质量数据集如何成为小语言模型获取数学推理能力的关键。我们提出\texttt{TinyGSM}——由GPT-3.5完全生成的1230万道小学数学题及其Python解答构成的合成数据集。对\texttt{TinyGSM}进行微调后，1.3B参数的生成模型与1.3B参数的验证模型组成的双模型系统可实现81.5%的准确率，超越参数规模大数个量级的现有模型。该性能甚至可与生成训练数据的GPT-3.5“教师”模型（77.4%）相媲美。该方法的简洁性体现在两个核心要素：1）高质量数据集\texttt{TinyGSM}；2）通过验证器从多个候选生成结果中筛选最终输出。