Although large language models (LLMs) has shown great performance on natural language processing (NLP) in the financial domain, there are no publicly available financial tailtored LLMs, instruction tuning datasets, and evaluation benchmarks, which is critical for continually pushing forward the open-source development of financial artificial intelligence (AI). This paper introduces PIXIU, a comprehensive framework including the first financial LLM based on fine-tuning LLaMA with instruction data, the first instruction data with 136K data samples to support the fine-tuning, and an evaluation benchmark with 5 tasks and 9 datasets. We first construct the large-scale multi-task instruction data considering a variety of financial tasks, financial document types, and financial data modalities. We then propose a financial LLM called FinMA by fine-tuning LLaMA with the constructed dataset to be able to follow instructions for various financial tasks. To support the evaluation of financial LLMs, we propose a standardized benchmark that covers a set of critical financial tasks, including five financial NLP tasks and one financial prediction task. With this benchmark, we conduct a detailed analysis of FinMA and several existing LLMs, uncovering their strengths and weaknesses in handling critical financial tasks. The model, datasets, benchmark, and experimental results are open-sourced to facilitate future research in financial AI.

翻译：尽管大语言模型（LLMs）在金融领域的自然语言处理（NLP）中展现出卓越性能，但目前仍缺乏公开可用的金融专用LLM、指令微调数据集及评估基准，而这对持续推动金融人工智能（AI）的开源发展至关重要。本文提出PIXIU，一个综合性框架，包含：首个基于LLaMA通过指令数据微调的金融LLM、首个含13.6万条样本以支持微调的指令数据集，以及涵盖5项任务与9个数据集的评估基准。我们首先构建了大规模多任务指令数据，综合考虑了多种金融任务、金融文档类型及金融数据模态。随后通过使用该数据集微调LLaMA，提出名为FinMA的金融LLM，使其能够遵循指令完成各类金融任务。为支撑金融LLM的评估，我们提出标准化基准，涵盖一组关键金融任务，包括五项金融NLP任务与一项金融预测任务。利用该基准，我们对FinMA及若干现有LLM进行详细分析，揭示其在处理关键金融任务时的优势与不足。相关模型、数据集、基准及实验结果均已开源，以促进金融AI领域的未来研究。