Large language models (LLMs) are increasingly pivotal in a wide range of natural language processing tasks. Access to pre-trained models, courtesy of the open-source community, has made it possible to adapt these models to specific applications for enhanced performance. However, the substantial resources required for training these models necessitate efficient solutions. This paper introduces CoLLiE, an efficient library that facilitates collaborative training of large language models using 3D parallelism, parameter-efficient fine-tuning (PEFT) methods, and optimizers such as Lion, Adan, Sophia, LOMO and AdaLomo. With its modular design and comprehensive functionality, CoLLiE offers a balanced blend of efficiency, ease of use, and customization. CoLLiE has proven superior training efficiency in comparison with prevalent solutions in pre-training and fine-tuning scenarios. Furthermore, we provide an empirical evaluation of the correlation between model size and GPU memory consumption under different optimization methods, as well as an analysis of the throughput. Lastly, we carry out a comprehensive comparison of various optimizers and PEFT methods within the instruction-tuning context. CoLLiE is available at https://github.com/OpenLMLab/collie.

翻译：大型语言模型（LLMs）在各类自然语言处理任务中日益发挥关键作用。得益于开源社区提供的预训练模型，研究人员能够将这些模型适配至特定应用场景以提升性能。然而，训练这些模型所需的大量资源促使我们寻求高效解决方案。本文介绍CoLLiE——一个高效库，通过三维并行、参数高效微调（PEFT）方法，以及Lion、Adan、Sophia、LOMO和AdaLomo等优化器，实现大型语言模型的协作训练。凭借其模块化设计及全面功能，CoLLiE在效率、易用性与自定义能力之间实现了均衡融合。在预训练与微调场景中，CoLLiE相较于主流方案展现出更优的训练效率。此外，我们实证评估了不同优化方法下模型规模与GPU内存消耗之间的关联性，并分析了吞吐量。最后，我们在指令微调场景中对多种优化器与PEFT方法进行了全面比较。CoLLiE已开源于https://github.com/OpenLMLab/collie。