The power of large language models (LLMs) has been demonstrated through numerous data and computing resources. However, the application of language models on mobile devices is facing huge challenge on the computation and memory costs, that is, tiny language models with high performance are urgently required. Limited by the highly complex training process, there are many details for optimizing language models that are seldom studied carefully. In this study, based on a tiny language model with 1B parameters, we carefully design a series of empirical study to analyze the effect of each component. Three perspectives are mainly discussed, i.e., neural architecture, parameter initialization, and optimization strategy. Several design formulas are empirically proved especially effective for tiny language models, including tokenizer compression, architecture tweaking, parameter inheritance and multiple-round training. Then we train PanGu-$\pi$-1B Pro and PanGu-$\pi$-1.5B Pro on 1.6T multilingual corpora, following the established formulas. Experimental results demonstrate the improved optimization and architecture yield a notable average improvement of 8.87 on benchmark evaluation sets for PanGu-$\pi$-1B Pro. Besides, PanGu-$\pi$-1.5B Pro surpasses a range of SOTA models with larger model sizes, validating its superior performance. The code will be released soon (https://github.com/YuchuanTian/RethinkTinyLM).

翻译：大型语言模型（LLMs）凭借海量数据和计算资源展示了其强大能力。然而，语言模型在移动设备上的应用面临计算和内存成本的巨大挑战，即迫切需要高性能的小型语言模型。受限于高度复杂的训练过程，语言模型优化中的许多细节鲜少得到细致研究。本研究基于一个参数规模为1B的小型语言模型，精心设计了一系列实证研究，以分析各组件的影响。主要从三个角度进行探讨：神经架构、参数初始化及优化策略。我们通过实验证明了若干设计公式对小型语言模型尤其有效，包括分词器压缩、架构调整、参数继承及多轮训练。随后，我们遵循这些公式，在1.6T多语言语料库上训练了PanGu-π-1B Pro和PanGu-π-1.5B Pro模型。实验结果表明，改进的优化与架构使PanGu-π-1B Pro在基准评测集上取得了平均8.87分的显著提升。此外，PanGu-π-1.5B Pro超越了一系列参数规模更大的最先进模型，验证了其卓越性能。代码即将开源（https://github.com/YuchuanTian/RethinkTinyLM）。