We introduce the Falcon series: 7B, 40B, and 180B parameters causal decoder-only models trained on a diverse high-quality corpora predominantly assembled from web data. The largest model, Falcon-180B, has been trained on over 3.5 trillion tokens of text--the largest openly documented pretraining run. Falcon-180B significantly outperforms models such as PaLM or Chinchilla, and improves upon concurrently developed models such as LLaMA 2 or Inflection-1. It nears the performance of PaLM-2-Large at a reduced pretraining and inference cost, making it, to our knowledge, one of the three best language models in the world along with GPT-4 and PaLM-2-Large. We report detailed evaluations, as well as a deep dive into the methods and custom tooling employed to pretrain Falcon. Notably, we report on our custom distributed training codebase, allowing us to efficiently pretrain these models on up to 4,096 A100s on cloud AWS infrastructure with limited interconnect. We release a 600B tokens extract of our web dataset, as well as the Falcon-7/40/180B models under a permissive license to foster open-science and accelerate the development of an open ecosystem of large language models.

翻译：我们推出Falcon系列模型：包含7B、40B和180B参数规模的因果解码器架构模型，这些模型主要基于从网络数据中筛选的高质量多样化语料库进行训练。其中最大模型Falcon-180B已在超过3.5万亿token文本上完成训练——这是目前公开文档记载中规模最大的预训练项目。Falcon-180B显著优于PaLM或Chinchilla等模型，并在同期开发的LLaMA 2或Inflection-1等模型基础上实现了性能突破。该模型以更低的预训练与推理成本逼近PaLM-2-Large的性能，据我们所知，这使得它与GPT-4和PaLM-2-Large并列成为全球三大最优语言模型。我们报告了详细的评估结果，并深入解析了用于预训练Falcon的方法与定制化工具。特别地，我们公开了定制化分布式训练代码库，使其能够在配备有限互联功能的AWS云基础设施上，利用最多4096个A100 GPU高效完成模型预训练。我们以宽松许可协议发布了包含600B token的网络数据集提取版，以及Falcon-7/40/180B系列模型，旨在推动开放科学并加速大语言模型开源生态系统的构建。