We introduce the Falcon series: 7B, 40B, and 180B parameters causal decoder-only models trained on a diverse high-quality corpora predominantly assembled from web data. The largest model, Falcon-180B, has been trained on over 3.5 trillion tokens of text--the largest openly documented pretraining run. Falcon-180B significantly outperforms models such as PaLM or Chinchilla, and improves upon concurrently developed models such as LLaMA 2 or Inflection-1. It nears the performance of PaLM-2-Large at a reduced pretraining and inference cost, making it, to our knowledge, one of the three best language models in the world along with GPT-4 and PaLM-2-Large. We report detailed evaluations, as well as a deep dive into the methods and custom tooling employed to pretrain Falcon. Notably, we report on our custom distributed training codebase, allowing us to efficiently pretrain these models on up to 4,096 A100s on cloud AWS infrastructure with limited interconnect. We release a 600B tokens extract of our web dataset, as well as the Falcon-7/40/180B models under a permissive license to foster open-science and accelerate the development of an open ecosystem of large language models.

翻译：我们推出Falcon系列模型：参数量分别为7B、40B和180B的因果解码器模型，基于以网络数据为主的多样化高质量语料库训练。其中最大模型Falcon-180B在超过3.5万亿文本token上完成训练——这是目前公开记录中规模最大的预训练过程。Falcon-180B在性能上显著超越PaLM、Chinchilla等模型，同时优于LLaMA 2、Inflection-1等同期开发模型。其性能接近PaLM-2-Large，但预训练与推理成本更低，据我们所知，该模型与GPT-4、PaLM-2-Large共同成为全球三大顶尖语言模型之一。本文报告了详细评估结果，并深度解析了预训练Falcon所采用的方法与定制工具。特别地，我们公开了自定义分布式训练代码库，使这些模型能够在配备有限互联的AWS云端基础设施上，最多使用4096块A100 GPU实现高效预训练。我们以宽松许可证形式发布了600B token的网络数据集子集及Falcon-7/40/180B模型权重，旨在推动开放科学并加速大语言模型开放生态的发展。