The derivation of mathematical results in specialised fields using Large Language Models (LLMs) is an emerging research direction that can help identify models' limitations, and potentially support mathematical discovery. In this paper, we leverage a symbolic engine to generate derivations of equations at scale, and investigate the capabilities of LLMs when deriving goal equations from premises. Specifically, we employ in-context learning for GPT and fine-tune a range of T5 models to compare the robustness and generalisation of pre-training strategies to specialised models. Empirical results show that fine-tuned FLAN-T5-large (MathT5) outperforms GPT models on all static and out-of-distribution test sets in terms of absolute performance. However, an in-depth analysis reveals that the fine-tuned models are more sensitive to perturbations involving unseen symbols and (to a lesser extent) changes to equation structure. In addition, we analyse 1.7K equations and over 200 derivations to highlight common reasoning errors such as the inclusion of incorrect, irrelevant, and redundant equations, along with the tendency to skip derivation steps. Finally, we explore the suitability of existing metrics for evaluating mathematical derivations finding evidence that, while they capture general properties such as sensitivity to perturbations, they fail to highlight fine-grained reasoning errors and essential differences between models. Overall, this work demonstrates that training models on synthetic data can improve their mathematical capabilities beyond larger architectures.

翻译：利用大型语言模型生成数学推导是新兴研究方向，有助于识别模型局限并潜在支持数学发现。本文利用符号引擎大规模生成方程推导，研究大型语言模型从前提推导目标方程的能力。具体而言，我们对GPT采用上下文学习，并微调一系列T5模型，以比较预训练策略对专用模型的鲁棒性和泛化能力。实验结果表明，微调后的FLAN-T5-large（MathT5）在静态和分布外测试集上的绝对性能均优于GPT模型。然而，深入分析显示，微调模型对未知符号的扰动更为敏感，对方程结构变化的敏感度次之。此外，我们分析1700个方程和200余条推导，揭示了常见推理错误，如包含错误、无关和冗余方程，以及跳过推导步骤的倾向。最后，我们探索了现有评估数学推导的指标适用性，发现尽管这些指标能捕捉扰动敏感度等通用属性，但无法凸显细粒度推理错误和模型间的本质差异。总体而言，本研究证明在合成数据上训练模型可提升其超越更大架构的数学能力。