The use of NLP in the realm of financial technology is broad and complex, with applications ranging from sentiment analysis and named entity recognition to question answering. Large Language Models (LLMs) have been shown to be effective on a variety of tasks; however, no LLM specialized for the financial domain has been reported in literature. In this work, we present BloombergGPT, a 50 billion parameter language model that is trained on a wide range of financial data. We construct a 363 billion token dataset based on Bloomberg's extensive data sources, perhaps the largest domain-specific dataset yet, augmented with 345 billion tokens from general purpose datasets. We validate BloombergGPT on standard LLM benchmarks, open financial benchmarks, and a suite of internal benchmarks that most accurately reflect our intended usage. Our mixed dataset training leads to a model that outperforms existing models on financial tasks by significant margins without sacrificing performance on general LLM benchmarks. Additionally, we explain our modeling choices, training process, and evaluation methodology. We release Training Chronicles (Appendix C) detailing our experience in training BloombergGPT.

翻译：自然语言处理在金融科技领域的应用广泛而复杂，涵盖从情感分析、命名实体识别到问答系统等多个任务。大型语言模型已被证明在多种任务中表现有效，然而，文献中尚未报道专门针对金融领域的大语言模型。本文提出BloombergGPT——一个基于海量金融数据训练的500亿参数语言模型。我们利用彭博的广泛数据源构建了3630亿词元的训练数据集，这可能是迄今为止最大的领域专用数据集，并辅以3450亿词元来自通用数据集。我们在标准大语言模型基准测试、公开金融基准测试以及最能反映我们实际使用场景的一系列内部基准测试上对BloombergGPT进行了验证。混合数据集训练使我们的模型在金融任务上显著优于现有模型，同时不失通用大语言模型基准测试的性能。此外，我们阐述了模型设计选择、训练过程及评估方法。我们还发布了训练日志（附录C），详细记录了BloombergGPT的训练经验。