Numerical reasoning is an important task in the analysis of financial documents. It helps in understanding and performing numerical predictions with logical conclusions for the given query seeking answers from financial texts. Recently, Large Language Models (LLMs) have shown promising results in multiple Question-Answering (Q-A) systems with the capability of logical reasoning. As documents related to finance often consist of long and complex financial contexts, LLMs appear well-suited for building high-quality automated financial question-answering systems. However, LLMs often face challenges in accurately processing the various numbers within financial reports. Extracting numerical data from unstructured text and semi-structured tables, and reliably performing accurate calculations, remains a significant bottleneck for numerical reasoning in most state-of-the-art LLMs. Recent studies have shown that structured data augmentations, such as Knowledge Graphs (KGs), have notably improved the predictions of LLMs along with logical explanations. Thus, it is an important requirement to consider inherent structured information in financial reports while using LLMs for various financial analytics. This paper proposes a framework to incorporate structured information using KGs along with LLM predictions for numerical reasoning tasks. The KGs are extracted using a proposed schema inherently from the document under processing. We evaluated our proposed framework over the benchmark data FinQA, using an open-source LLM, namely Llama 3.1 8B Instruct. We observed that the proposed framework improved execution accuracy by approximately 12% relative to the vanilla LLM.

翻译：数值推理是金融文档分析中的一项重要任务。它有助于理解金融文本并基于逻辑结论进行数值预测，以回答给定查询。近年来，大语言模型（LLMs）凭借其逻辑推理能力，在多个问答系统中展现出优异性能。由于金融相关文档通常包含冗长且复杂的金融语境，LLMs 似乎非常适合构建高质量的自动化金融问答系统。然而，LLMs 在处理财务报表中的各类数字时常常面临挑战。从非结构化文本和半结构化表格中提取数值数据，并可靠地执行精确计算，仍然是当前大多数先进 LLMs 在数值推理方面的重要瓶颈。近期研究表明，知识图谱（KGs）等结构化数据增强方法显著提升了 LLMs 的预测能力及其逻辑解释性。因此，在使用 LLMs 进行各类金融分析时，充分考虑财务报表中固有的结构化信息成为一项重要需求。本文提出一个框架，将基于 KGs 的结构化信息与 LLM 预测相结合，用于数值推理任务。KGs 通过提出的模式从处理文档中固有地提取。我们在基准数据集 FinQA 上使用开源 LLM（即 Llama 3.1 8B Instruct）评估了所提出的框架。实验结果表明，相较于原始 LLM，该框架将执行准确率相对提升了约 12%。