Large Language Models (LLMs) have demonstrated remarkable performance across various natural language tasks, marking significant strides towards general artificial intelligence. While general artificial intelligence is leveraged by developing increasingly large-scale models, there could be another branch to develop lightweight custom models that better serve certain domains, taking into account the high cost of training and deploying LLMs and the scarcity of resources. In this paper, we present MindLLM, a novel series of bilingual lightweight large language models, trained from scratch, alleviating such burdens by offering models with 1.3 billion and 3 billion parameters. A thorough account of experiences accrued during large model development is given, covering every step of the process, including data construction, model architecture, evaluation, and applications. Such insights are hopefully valuable for fellow academics and developers. MindLLM consistently matches or surpasses the performance of other open-source larger models on some public benchmarks. We also introduce an innovative instruction tuning framework tailored for smaller models to enhance their capabilities efficiently. Moreover, we explore the application of MindLLM in specific vertical domains such as law and finance, underscoring the agility and adaptability of our lightweight models.

翻译：大语言模型（LLMs）在各类自然语言任务中展现出卓越性能，标志着向通用人工智能迈出了重要步伐。尽管通用人工智能通过日益庞大的模型规模得以实现，但考虑到训练和部署大语言模型的高昂成本以及资源稀缺性，另一发展方向或许是开发能更好服务于特定领域的轻量级定制模型。本文提出MindLLM系列——从零开始训练的新型双语轻量级大语言模型，通过提供13亿和30亿参数的模型来减轻上述负担。我们详细阐述了大型模型开发过程中积累的经验，涵盖数据构建、模型架构、评估与应用等全流程环节。这些见解对学界同行和开发者具有重要参考价值。MindLLM在多项公开基准测试中持续达到或超越其他开源大型模型的性能表现。我们还创新性地提出专为小模型设计的指令调优框架，以高效提升其能力。此外，我们探索了MindLLM在法律、金融等特定垂直领域的应用，充分彰显了轻量级模型的敏捷性与适配性。