Chain-of-thought (CoT) prompting with large language models has proven effective in numerous natural language processing tasks, but designing prompts that generalize well to diverse problem types can be challenging, especially in the context of math word problem (MWP) solving. Additionally, it is common to have a large amount of training data that have a better diversity coverage but CoT annotations are not available, which limits the use of supervised learning techniques. To address these issues, we investigate two approaches to leverage the training data in a few-shot prompting scenario: dynamic program prompting and program distillation. Our approach is largely inspired by Gao et al., (2022), where they proposed to replace the CoT with the programs as the intermediate reasoning step. Such a prompting strategy allows us to accurately verify the answer correctness through program execution in MWP solving. Our dynamic program prompting involves annotating the training data by sampling correct programs from a large language model, while program distillation involves adapting a smaller model to the program-annotated training data. Our experiments on three standard MWP datasets demonstrate the effectiveness of these approaches, yielding significant improvements over previous baselines for prompting and fine-tuning. Our results suggest that leveraging a large amount of training data can improve the generalization ability of prompts and boost the performance of fine-tuned small models in MWP solving.

翻译：链式思维（CoT）提示结合大语言模型已在众多自然语言处理任务中展现出有效性，但设计能良好泛化至不同类型问题的提示仍具挑战性，尤其在数学文字题（MWP）求解场景中。此外，实践中常存在大量训练数据（具备更优多样性覆盖范围）却缺乏CoT标注，这限制了监督学习技术的应用。为解决上述问题，我们探究了两种在小样本提示场景中利用训练数据的方法：动态程序提示与程序蒸馏。本方法深受Gao等人（2022）启发——他们提出以程序替代CoT作为中间推理步骤。这种提示策略使我们能通过程序执行精确验证MWP求解中答案的正确性。动态程序提示通过从大语言模型中采样正确程序来标注训练数据，而程序蒸馏则使小型模型适应程序标注后的训练数据。在三个标准MWP数据集上的实验表明，这些方法效果显著，相较于以往的提示学习和微调基线方法取得大幅提升。研究结果表明，利用大规模训练数据既能增强提示的泛化能力，也能提升微调小模型在MWP求解中的性能表现。