In this study, we introduce CT-LLM, a 2B large language model (LLM) that illustrates a pivotal shift towards prioritizing the Chinese language in developing LLMs. Uniquely initiated from scratch, CT-LLM diverges from the conventional methodology by primarily incorporating Chinese textual data, utilizing an extensive corpus of 1,200 billion tokens, including 800 billion Chinese tokens, 300 billion English tokens, and 100 billion code tokens. This strategic composition facilitates the model's exceptional proficiency in understanding and processing Chinese, a capability further enhanced through alignment techniques. Demonstrating remarkable performance on the CHC-Bench, CT-LLM excels in Chinese language tasks, and showcases its adeptness in English through SFT. This research challenges the prevailing paradigm of training LLMs predominantly on English corpora and then adapting them to other languages, broadening the horizons for LLM training methodologies. By open-sourcing the full process of training a Chinese LLM, including a detailed data processing procedure with the obtained Massive Appropriate Pretraining Chinese Corpus (MAP-CC), a well-chosen multidisciplinary Chinese Hard Case Benchmark (CHC-Bench), and the 2B-size Chinese Tiny LLM (CT-LLM), we aim to foster further exploration and innovation in both academia and industry, paving the way for more inclusive and versatile language models.

翻译：在本研究中，我们提出了CT-LLM，一个20亿参数的大型语言模型，其核心转向了以中文为中心的语言模型开发。与常规方法不同，CT-LLM从零开始，主要采用中文文本数据进行训练，使用了包含1.2万亿个token的庞大数据集，其中包括8000亿个中文token、3000亿个英文token和1000亿个代码token。这种策略性的数据组合使模型在处理和理解中文方面表现出卓越能力，并通过对齐技术进一步强化。在CHC-Bench基准测试中，CT-LLM在中文任务上表现出色，同时通过监督微调展示了其英语处理能力。本研究挑战了主流训练范式——即主要基于英文语料训练语言模型再迁移至其他语言，拓宽了语言模型训练方法的视野。通过开源中文语言模型的完整训练流程（包括基于获得的大规模适切中文预训练语料库的数据处理详细流程、精心选择的多学科中文硬案例基准测试集以及20亿参数的中文小语言模型），我们旨在促进学术界与工业界的探索创新，为更具包容性和多功能的语言模型铺平道路。