Large Language Models (LLMs) have demonstrated remarkable performance across various natural language tasks, marking significant strides towards general artificial intelligence. While general artificial intelligence is leveraged by developing increasingly large-scale models, there could be another branch to develop lightweight custom models that better serve certain domains, taking into account the high cost of training and deploying LLMs and the scarcity of resources. In this paper, we present MindLLM, a novel series of bilingual lightweight large language models, trained from scratch, alleviating such burdens by offering models with 1.3 billion and 3 billion parameters. A thorough account of experiences accrued during large model development is given, covering every step of the process, including data construction, model architecture, evaluation, and applications. Such insights are hopefully valuable for fellow academics and developers. MindLLM consistently matches or surpasses the performance of other open-source larger models on some public benchmarks. We also introduce an innovative instruction tuning framework tailored for smaller models to enhance their capabilities efficiently. Moreover, we explore the application of MindLLM in specific vertical domains such as law and finance, underscoring the agility and adaptability of our lightweight models.

翻译：大语言模型（LLMs）在各类自然语言任务中展现出卓越性能，标志着向通用人工智能迈出了重要步伐。尽管通用人工智能通过开发日益大规模的模型而得以实现，但考虑到训练和部署LLMs的高昂成本以及资源稀缺性，另一条发展路径可能是开发轻量级定制模型以更好地服务于特定领域。本文提出MindLLM系列——从零训练的1.3B与3B参数双语轻量级大语言模型，旨在缓解上述成本与资源压力。我们系统阐述了大型模型开发全流程的实践经验，涵盖数据构建、模型架构、评估与应用等各环节。这些洞见或将为学术界与开发者提供宝贵参考。MindLLM在多项公开基准测试中持续达到或超越其他开源大型模型的性能。我们还创新性地提出专为小型模型设计的指令微调框架，有效提升其能力。此外，我们探索了MindLLM在法律、金融等垂直领域的应用，充分彰显轻量级模型的灵活性与适应性。