The burgeoning interest in developing Large Language Models (LLMs) with up to trillion parameters has been met with concerns regarding resource efficiency and practical expense, particularly given the immense cost of experimentation. This scenario underscores the importance of exploring the potential of Small Language Models (SLMs) as a resource-efficient alternative. In this context, we introduce MiniCPM, specifically the 1.2B and 2.4B non-embedding parameter variants, not only excel in their respective categories but also demonstrate capabilities on par with 7B-13B LLMs. While focusing on SLMs, our approach exhibits scalability in both model and data dimensions for future LLM research. Regarding model scaling, we employ extensive model wind tunnel experiments for stable and optimal scaling. For data scaling, we introduce a Warmup-Stable-Decay (WSD) learning rate scheduler (LRS), conducive to continuous training and domain adaptation. We present an in-depth analysis of the intriguing training dynamics that occurred in the WSD LRS. With WSD LRS, we are now able to efficiently study data-model scaling law without extensive retraining experiments on both axes of model and data, from which we derive the much higher compute optimal data-model ratio than Chinchilla Optimal. Additionally, we introduce MiniCPM family, including MiniCPM-DPO, MiniCPM-MoE and MiniCPM-128K, whose excellent performance further cementing MiniCPM's foundation in diverse SLM applications. MiniCPM models are available publicly at https://github.com/OpenBMB/MiniCPM .

翻译：随着开发参数规模高达万亿级别的大型语言模型（LLMs）的兴趣日益高涨，资源效率和实际成本问题也引发了广泛关注，尤其是在实验成本极其高昂的背景下。这一现状凸显了探索小型语言模型（SLMs）作为资源高效替代方案的重要性。在此背景下，我们推出MiniCPM，特别是其1.2B和2.4B非嵌入参数变体，这些模型不仅在各自类别中表现卓越，还展现出与7B-13B参数规模的LLMs相媲美的能力。尽管聚焦于SLMs，我们的方法在模型和数据维度上均展现出可扩展性，为未来的LLM研究提供了参考。在模型扩展方面，我们通过大量模型风洞实验实现稳定且最优的扩展。在数据扩展方面，我们引入了预热-稳定-衰减（WSD）学习率调度器（LRS），该调度器有利于持续训练和领域适应。我们对WSD LRS中出现的引人入胜的训练动态进行了深入分析。借助WSD LRS，我们现在能够高效地研究数据-模型缩放规律，而无需在模型和数据两个维度上进行大量重复训练实验，并由此推导出远高于Chinchilla最优值的计算最优数据-模型比例。此外，我们推出了MiniCPM系列模型，包括MiniCPM-DPO、MiniCPM-MoE和MiniCPM-128K，其卓越性能进一步巩固了MiniCPM在多样化SLM应用中的基础地位。MiniCPM模型已在https://github.com/OpenBMB/MiniCPM 公开提供。