The power of large language models (LLMs) has been demonstrated through numerous data and computing resources. However, the application of language models on mobile devices is facing huge challenge on the computation and memory costs, that is, tiny language models with high performance are urgently required. Limited by the highly complex training process, there are many details for optimizing language models that are seldom studied carefully. In this study, based on a tiny language model with 1B parameters, we carefully design a series of empirical study to analyze the effect of each component. Three perspectives are mainly discussed, \ie, neural architecture, parameter initialization, and optimization strategy. Several design formulas are empirically proved especially effective for tiny language models, including tokenizer compression, architecture tweaking, parameter inheritance and multiple-round training. Then we train PanGu-$\pi$-1B Pro and PanGu-$\pi$-1.5B Pro on 1.6T multilingual corpora, following the established formulas. Experimental results demonstrate the improved optimization and architecture yield a notable average improvement of 8.87 on benchmark evaluation sets for PanGu-$\pi$-1B Pro. Besides, PanGu-$\pi$-1.5B Pro surpasses a range of SOTA models with larger model sizes, validating its superior performance. The code is available at https://github.com/YuchuanTian/RethinkTinyLM.

翻译：大型语言模型（LLMs）的强大能力已通过海量数据和计算资源得到充分证明。然而，语言模型在移动设备上的应用正面临计算和存储成本的巨大挑战，即迫切需要高性能的小型语言模型。受限于高度复杂的训练过程，许多优化语言模型的细节尚未得到充分研究。本研究基于一个1B参数的小型语言模型，精心设计了一系列实证研究来分析每个组件的影响。主要从三个角度进行讨论，即神经架构、参数初始化和优化策略。多种设计公式被实证证明对小型语言模型尤其有效，包括分词器压缩、架构调整、参数继承和多轮训练。随后，我们遵循这些既定公式，在1.6T多语言语料上训练了PanGu-π-1B Pro和PanGu-π-1.5B Pro。实验结果表明，改进的优化和架构使PanGu-π-1B Pro在基准评估集上取得了平均8.87分的显著提升。此外，PanGu-π-1.5B Pro以更小的模型规模超越了多种更大规模的最先进（SOTA）模型，验证了其卓越性能。代码已开源至https://github.com/YuchuanTian/RethinkTinyLM。