Recently, many works have proposed various financial large language models (FinLLMs) by pre-training from scratch or fine-tuning open-sourced LLMs on financial corpora. However, existing FinLLMs exhibit unsatisfactory performance in understanding financial text when numeric variables are involved in questions. In this paper, we propose a novel LLM, called numeric-sensitive large language model (NumLLM), for Chinese finance. We first construct a financial corpus from financial textbooks which is essential for improving numeric capability of LLMs during fine-tuning. After that, we train two individual low-rank adaptation (LoRA) modules by fine-tuning on our constructed financial corpus. One module is for adapting general-purpose LLMs to financial domain, and the other module is for enhancing the ability of NumLLM to understand financial text with numeric variables. Lastly, we merge the two LoRA modules into the foundation model to obtain NumLLM for inference. Experiments on financial question-answering benchmark show that NumLLM can boost the performance of the foundation model and can achieve the best overall performance compared to all baselines, on both numeric and non-numeric questions.

翻译：摘要：近期，许多研究通过从零开始预训练或在金融语料上微调开源大语言模型，提出了多种金融大语言模型（FinLLMs）。然而，当问题涉及数值变量时，现有FinLLMs在理解金融文本方面的表现仍不尽人意。本文针对中文金融领域，提出了一种名为数值敏感型大语言模型（NumLLM）的新型模型。我们首先从金融教材中构建金融语料库，这对微调过程中提升LLMs的数值能力至关重要。随后，我们基于所构建的金融语料库，分别训练两个独立的低秩适应（LoRA）模块：一个模块用于将通用大语言模型适配到金融领域，另一个模块用于增强NumLLM理解含数值变量的金融文本的能力。最后，我们将两个LoRA模块合并至基础模型中，得到用于推理的NumLLM。在金融问答基准上的实验表明，NumLLM能够提升基础模型的性能，并在数值问题与非数值问题上均能取得优于所有基线模型的整体最佳表现。