The recent surge in open-source Large Language Models (LLMs), such as LLaMA, Falcon, and Mistral, provides diverse options for AI practitioners and researchers. However, most LLMs have only released partial artifacts, such as the final model weights or inference code, and technical reports increasingly limit their scope to high-level design choices and surface statistics. These choices hinder progress in the field by degrading transparency into the training of LLMs and forcing teams to rediscover many details in the training process. We present LLM360, an initiative to fully open-source LLMs, which advocates for all training code and data, model checkpoints, and intermediate results to be made available to the community. The goal of LLM360 is to support open and collaborative AI research by making the end-to-end LLM training process transparent and reproducible by everyone. As a first step of LLM360, we release two 7B parameter LLMs pre-trained from scratch, Amber and CrystalCoder, including their training code, data, intermediate checkpoints, and analyses (at https://www.llm360.ai). We are committed to continually pushing the boundaries of LLMs through this open-source effort. More large-scale and stronger models are underway and will be released in the future.

翻译：近期开源大语言模型（LLMs）的蓬勃发展，例如LLaMA、Falcon和Mistral，为人工智能从业者和研究者提供了多样化的选择。然而，大多数大语言模型仅发布了部分成果，如最终模型权重或推理代码，且技术报告日益局限于高层设计选择和粗略统计数据。这些做法削弱了大语言模型训练过程的透明度，迫使研究团队在训练流程中重新探索诸多细节，从而阻碍了该领域的进步。我们提出LLM360倡议，致力于完全开源大语言模型，主张将全部训练代码与数据、模型检查点以及中间结果向社区开放。LLM360的目标是通过使端到端的大语言模型训练过程透明化且可被任何人复现，从而支持开放协作式的人工智能研究。作为LLM360的第一步，我们发布了两个从头开始预训练的70亿参数大语言模型——Amber和CrystalCoder，包括其训练代码、数据、中间检查点及分析（详见https://www.llm360.ai）。我们承诺持续通过此项开源工作推动大语言模型的边界拓展。更多大规模且性能更强的模型正在研发中，将于未来陆续发布。