Financial decision-making hinges on the analysis of relevant information embedded in the enormous volume of documents in the financial domain. To address this challenge, we developed FinQAPT, an end-to-end pipeline that streamlines the identification of relevant financial reports based on a query, extracts pertinent context, and leverages Large Language Models (LLMs) to perform downstream tasks. To evaluate the pipeline, we experimented with various techniques to optimize the performance of each module using the FinQA dataset. We introduced a novel clustering-based negative sampling technique to enhance context extraction and a novel prompting method called Dynamic N-shot Prompting to boost the numerical question-answering capabilities of LLMs. At the module level, we achieved state-of-the-art accuracy on FinQA, attaining an accuracy of 80.6%. However, at the pipeline level, we observed decreased performance due to challenges in extracting relevant context from financial reports. We conducted a detailed error analysis of each module and the end-to-end pipeline, pinpointing specific challenges that must be addressed to develop a robust solution for handling complex financial tasks.

翻译：金融决策依赖于对金融领域海量文档中嵌入的相关信息进行分析。为应对这一挑战，我们开发了FinQAPT——一个端到端流程，能够根据查询高效识别相关财务报告、提取关联上下文，并利用大语言模型执行下游任务。为评估该流程，我们采用FinQA数据集实验了多种技术以优化各模块性能。我们提出了一种新颖的基于聚类的负采样技术以增强上下文提取能力，并设计了名为动态N样本提示的新型提示方法以提升LLM的数值问答能力。在模块层面，我们在FinQA数据集上取得了80.6%准确率的最优性能。然而在流程层面，由于从财务报告中提取相关上下文存在挑战，我们观察到性能有所下降。我们对各模块及端到端流程进行了详细的错误分析，明确了开发稳健解决方案以处理复杂金融任务必须应对的具体挑战。