Mathematical reasoning, a core ability of human intelligence, presents unique challenges for machines in abstract thinking and logical reasoning. Recent large pre-trained language models such as GPT-3 have achieved remarkable progress on mathematical reasoning tasks written in text form, such as math word problems (MWP). However, it is unknown if the models can handle more complex problems that involve math reasoning over heterogeneous information, such as tabular data. To fill the gap, we present Tabular Math Word Problems (TabMWP), a new dataset containing 38,431 open-domain grade-level problems that require mathematical reasoning on both textual and tabular data. Each question in TabMWP is aligned with a tabular context, which is presented as an image, semi-structured text, and a structured table. There are two types of questions: free-text and multi-choice, and each problem is annotated with gold solutions to reveal the multi-step reasoning process. We evaluate different pre-trained models on TabMWP, including the GPT-3 model in a few-shot setting. As earlier studies suggest, since few-shot GPT-3 relies on the selection of in-context examples, its performance is unstable and can degrade to near chance. The unstable issue is more severe when handling complex problems like TabMWP. To mitigate this, we further propose a novel approach, PromptPG, which utilizes policy gradient to learn to select in-context examples from a small amount of training data and then constructs the corresponding prompt for the test example. Experimental results show that our method outperforms the best baseline by 5.31% on the accuracy metric and reduces the prediction variance significantly compared to random selection, which verifies its effectiveness in selecting in-context examples.

翻译：数学推理作为人类智能的核心能力，为机器在抽象思维与逻辑推理方面带来了独特挑战。近期，诸如GPT-3等大规模预训练语言模型在文本形式的数学推理任务（如数学应用题）上取得了显著进展。然而，这些模型能否处理涉及表格数据等异构信息进行数学推理的更复杂问题，仍尚不明确。为填补这一空白，我们提出表格数学应用题（TabMWP）数据集，包含38,431个面向开放领域的年级水平问题，需同时对文本和表格数据进行数学推理。TabMWP中的每个问题均与表格语境对齐，该语境以图像、半结构化文本和结构化表格三种形式呈现。问题类型分为自由文本题与多项选择题两类，每个问题均附有标准答案以揭示多步推理过程。我们在TabMWP上评估了不同预训练模型（包括少样本场景下的GPT-3模型）。正如早期研究所指出的，由于少样本GPT-3依赖于上下文示例的选择，其性能不稳定，甚至可能退化至随机水平。这种不稳定性在处理TabMWP等复杂问题时更为严重。为此，我们进一步提出创新方法PromptPG，利用策略梯度从少量训练数据中学习选择上下文示例，进而为测试样本构建相应提示。实验结果表明，该方法在准确率指标上超越最佳基线5.31%，与随机选择相比显著降低了预测方差，验证了其在上下文示例选择中的有效性。