Large Language Models (LLMs), such as ChatGPT and GPT-4, have dramatically transformed natural language processing research and shown promising strides towards Artificial General Intelligence (AGI). Nonetheless, the high costs associated with training and deploying LLMs present substantial obstacles to transparent, accessible academic research. While several large language models, such as LLaMA, have been open-sourced by the community, these predominantly focus on English corpora, limiting their usefulness for other languages. In this paper, we propose a method to augment LLaMA with capabilities for understanding and generating Chinese text and its ability to follow instructions. We achieve this by extending LLaMA's existing vocabulary with an additional 20,000 Chinese tokens, thereby improving its encoding efficiency and semantic understanding of Chinese. We further incorporate secondary pre-training using Chinese data and fine-tune the model with Chinese instruction datasets, significantly enhancing the model's ability to comprehend and execute instructions. Our experimental results indicate that the newly proposed model markedly enhances the original LLaMA's proficiency in understanding and generating Chinese content. Additionally, the results on the C-Eval dataset yield competitive performance among the models with several times the size of ours. We have made our pre-trained models, training scripts, and other resources available through GitHub, fostering open research for our community. GitHub repository: https://github.com/ymcui/Chinese-LLaMA-Alpaca

翻译：大型语言模型（LLMs），如ChatGPT和GPT-4，已深刻改变了自然语言处理研究，并向通用人工智能（AGI）迈出了显著步伐。然而，训练和部署LLMs的高昂成本为透明、可获取的学术研究带来了重大障碍。尽管社区已开源了多个大型语言模型（例如LLaMA），但这些模型主要侧重于英语语料库，限制了其对其他语言的适用性。本文提出一种方法，用于增强LLaMA理解和生成中文文本以及遵循指令的能力。我们通过向LLaMA现有词汇表扩展20,000个中文标记来实现这一目标，从而提升其对中文的编码效率和语义理解。我们进一步利用中文数据进行二次预训练，并使用中文指令数据集对模型进行微调，显著增强了模型理解并执行指令的能力。实验结果表明，新提出的模型显著提升了原始LLaMA在理解和生成中文内容方面的熟练度。此外，在C-Eval数据集上的结果显示出与规模数倍于我们的模型相比具有竞争力的性能。我们已将预训练模型、训练脚本及其他资源通过GitHub公开，以促进社区的开放式研究。GitHub仓库: https://github.com/ymcui/Chinese-LLaMA-Alpaca