The use of NLP in the realm of financial technology is broad and complex, with applications ranging from sentiment analysis and named entity recognition to question answering. Large Language Models (LLMs) have been shown to be effective on a variety of tasks; however, no LLM specialized for the financial domain has been reported in literature. In this work, we present BloombergGPT, a 50 billion parameter language model that is trained on a wide range of financial data. We construct a 363 billion token dataset based on Bloomberg's extensive data sources, perhaps the largest domain-specific dataset yet, augmented with 345 billion tokens from general purpose datasets. We validate BloombergGPT on standard LLM benchmarks, open financial benchmarks, and a suite of internal benchmarks that most accurately reflect our intended usage. Our mixed dataset training leads to a model that outperforms existing models on financial tasks by significant margins without sacrificing performance on general LLM benchmarks. Additionally, we explain our modeling choices, training process, and evaluation methodology. As a next step, we plan to release training logs (Chronicles) detailing our experience in training BloombergGPT.

翻译：自然语言处理在金融科技领域的应用广泛且复杂，涵盖从情感分析、命名实体识别到问答系统等多项任务。大型语言模型已被证明在多种任务上表现出色，然而文献中尚未报道过专门针对金融领域的大型语言模型。本文提出BloombergGPT——一个基于广泛金融数据训练的500亿参数语言模型。我们利用彭博的海量数据源构建了包含3630亿词元的训练集（这或许是迄今为止最大的领域专用数据集），并辅以3450亿词元来自通用数据集。我们在标准大型语言模型基准测试、公开金融基准测试以及最能反映其预期用途的内部基准测试套件上对BloombergGPT进行了验证。混合数据集训练使模型在金融任务上的表现显著优于现有模型，且未牺牲通用大型语言模型基准测试的性能。此外，我们还阐释了建模选择、训练过程及评估方法。下一步计划公开发布记录BloombergGPT训练过程的训练日志（编年史）。