In this study, we introduce CT-LLM, a 2B large language model (LLM) that illustrates a pivotal shift towards prioritizing the Chinese language in developing LLMs. Uniquely initiated from scratch, CT-LLM diverges from the conventional methodology by primarily incorporating Chinese textual data, utilizing an extensive corpus of 1,200 billion tokens, including 800 billion Chinese tokens, 300 billion English tokens, and 100 billion code tokens. This strategic composition facilitates the model's exceptional proficiency in understanding and processing Chinese, a capability further enhanced through alignment techniques. Demonstrating remarkable performance on the CHC-Bench, CT-LLM excels in Chinese language tasks, and showcases its adeptness in English through SFT. This research challenges the prevailing paradigm of training LLMs predominantly on English corpora and then adapting them to other languages, broadening the horizons for LLM training methodologies. By open-sourcing the full process of training a Chinese LLM, including a detailed data processing procedure with the obtained Massive Appropriate Pretraining Chinese Corpus (MAP-CC), a well-chosen multidisciplinary Chinese Hard Case Benchmark (CHC-Bench), and the 2B-size Chinese Tiny LLM (CT-LLM), we aim to foster further exploration and innovation in both academia and industry, paving the way for more inclusive and versatile language models.

翻译：本研究提出了CT-LLM，一个20亿参数的大语言模型，彰显了在开发大语言模型时优先关注中文的重大转向。该模型创新性地从零开始训练，摒弃了传统方法，主要采用中文文本数据，使用了包含1.2万亿个token的大规模语料库，其中含8000亿中文token、3000亿英文token和1000亿代码token。这种战略性语料构成使模型在理解和处理中文方面展现出卓越能力，并通过对齐技术进一步强化。在CHC-Bench基准测试中，CT-LLM在中文语言任务上表现优异，同时通过监督微调展现了其在英文任务上的熟练度。本研究挑战了当前主要基于英文语料训练大语言模型再迁移至其他语言的范式，拓展了大语言模型训练方法的视野。通过开源中文大语言模型的完整训练流程，包括基于所获取的海量适配中文预训练语料库的详细数据处理流程、精心选择的多学科中文硬案例基准以及20亿参数的中文小模型，我们旨在促进学术界和工业界的进一步探索与创新，为更具包容性和通用性的语言模型铺平道路。