The derivation of mathematical results in specialised fields using Large Language Models (LLMs) is an emerging research direction that can help identify models' limitations, and potentially support mathematical discovery. In this paper, we leverage a symbolic engine to generate derivations of equations at scale, and investigate the capabilities of LLMs when deriving goal equations from premises. Specifically, we employ in-context learning for GPT and fine-tune a range of T5 models to compare the robustness and generalisation of pre-training strategies to specialised models. Empirical results show that fine-tuned FLAN-T5-large (MathT5) outperforms GPT models on all static and out-of-distribution test sets in terms of absolute performance. However, an in-depth analysis reveals that the fine-tuned models are more sensitive to perturbations involving unseen symbols and (to a lesser extent) changes to equation structure. In addition, we analyse 1.7K equations and over 200 derivations to highlight common reasoning errors such as the inclusion of incorrect, irrelevant, and redundant equations, along with the tendency to skip derivation steps. Finally, we explore the suitability of existing metrics for evaluating mathematical derivations finding evidence that, while they capture general properties such as sensitivity to perturbations, they fail to highlight fine-grained reasoning errors and essential differences between models. Overall, this work demonstrates that training models on synthetic data can improve their mathematical capabilities beyond larger architectures.

翻译：专业领域中利用大型语言模型（LLMs）生成数学结果是一个新兴研究方向，有助于揭示模型局限性并可能支持数学发现。本文利用符号引擎大规模生成方程推导过程，探究LLM从前提推导目标方程的能力。具体而言，我们采用GPT的上下文学习策略，并微调一系列T5模型，以比较预训练策略在专用模型中的鲁棒性与泛化能力。实验结果表明，微调后的FLAN-T5-large（MathT5）在所有静态与分布外测试集上的绝对性能均优于GPT模型。然而，深入分析显示，微调模型对涉及未见符号的扰动更为敏感，对等式结构变化的敏感度较低。此外，我们分析1,700个方程及200余组推导过程，揭示了常见推理错误类型，包括引入错误、无关和冗余方程，以及跳过推导步骤的倾向。最后，我们评估了现有数学推导评价指标的适用性，发现其虽能捕捉对扰动的敏感性等通用特性，却未能凸显细粒度推理错误与模型间本质差异。总体而言，本研究证明基于合成数据训练的模型可超越更大架构的数学推理能力。