Solving symbolic mathematics has always been of in the arena of human ingenuity that needs compositional reasoning and recurrence. However, recent studies have shown that large-scale language models such as transformers are universal and surprisingly can be trained as a sequence-to-sequence task to solve complex mathematical equations. These large transformer models need humongous amounts of training data to generalize to unseen symbolic mathematics problems. In this paper, we present a sample efficient way of solving the symbolic tasks by first pretraining the transformer model with language translation and then fine-tuning the pretrained transformer model to solve the downstream task of symbolic mathematics. We achieve comparable accuracy on the integration task with our pretrained model while using around $1.5$ orders of magnitude less number of training samples with respect to the state-of-the-art deep learning for symbolic mathematics. The test accuracy on differential equation tasks is considerably lower comparing with integration as they need higher order recursions that are not present in language translations. We propose the generalizability of our pretrained language model from Anna Karenina Principle (AKP). We pretrain our model with different pairs of language translations. Our results show language bias in solving symbolic mathematics tasks. Finally, we study the robustness of the fine-tuned model on symbolic math tasks against distribution shift, and our approach generalizes better in distribution shift scenarios for the function integration.

翻译：解决符号数学问题历来属于人类智慧的范畴，需要组合推理与递归能力。然而，近年研究表明，Transformer等大规模语言模型具有通用性，且令人惊讶地能够通过序列到序列任务训练来解决复杂数学方程。这些大型Transformer模型需要海量训练数据才能泛化至未见过的符号数学问题。本文提出一种样本高效的符号任务求解方法：首先用语言翻译任务预训练Transformer模型，再微调该预训练模型以完成下游符号数学任务。在积分任务上，我们使用比当前最先进的符号数学深度学习方法约1.5个数量级的训练样本，即可达到相当精度。而微分方程任务的测试精度显著低于积分任务，因其需要语言翻译中不存在的更高阶递归。我们从安娜·卡列尼娜原理（AKP）出发论证了预训练语言模型的泛化能力。通过使用不同语言翻译对进行预训练，结果表明语言偏差会影响符号数学任务的求解。最后，我们研究了微调模型在符号数学任务中应对分布偏移的鲁棒性，本方法在函数积分场景下具有更好的分布偏移泛化性能。