In this study, we introduce CT-LLM, a 2B large language model (LLM) that illustrates a pivotal shift towards prioritizing the Chinese language in developing LLMs. Uniquely initiated from scratch, CT-LLM diverges from the conventional methodology by primarily incorporating Chinese textual data, utilizing an extensive corpus of 1,200 billion tokens, including 800 billion Chinese tokens, 300 billion English tokens, and 100 billion code tokens. This strategic composition facilitates the model's exceptional proficiency in understanding and processing Chinese, a capability further enhanced through alignment techniques. Demonstrating remarkable performance on the CHC-Bench, CT-LLM excels in Chinese language tasks, and showcases its adeptness in English through SFT. This research challenges the prevailing paradigm of training LLMs predominantly on English corpora and then adapting them to other languages, broadening the horizons for LLM training methodologies. By open-sourcing the full process of training a Chinese LLM, including a detailed data processing procedure with the obtained Massive Appropriate Pretraining Chinese Corpus (MAP-CC), a well-chosen multidisciplinary Chinese Hard Case Benchmark (CHC-Bench), and the 2B-size Chinese Tiny LLM (CT-LLM), we aim to foster further exploration and innovation in both academia and industry, paving the way for more inclusive and versatile language models.

翻译：本研究提出CT-LLM，一个20亿参数的大语言模型，标志着大语言模型开发向优先关注中文的重要转变。该模型从零开始训练，不同于传统方法，主要通过纳入中文文本数据，使用了包含1.2万亿个标记的大规模语料库，其中包含8000亿中文标记、3000亿英文标记和1000亿代码标记。这种策略性组成使模型在理解和处理中文方面具有卓越能力，并通过对齐技术进一步增强。在CHC-Bench基准测试中，CT-LLM在中文任务中表现优异，并通过监督微调展现了其处理英文文本的熟练度。本研究挑战了主要基于英文语料训练大语言模型再适配其他语言的传统范式，拓展了大语言模型训练方法的视野。通过开源完整训练中文大语言模型的全流程，包括详细的数据处理流程（所获大规模适切中文预训练语料库MAP-CC）、精心挑选的多学科中文硬案例基准测试（CHC-Bench）以及20亿参数规模的中文小模型（CT-LLM），我们旨在促进学术界和工业界的进一步探索与创新，为构建更具包容性和多功能性的语言模型铺平道路。