Large Language Models (LLMs) have achieved remarkable results. However, existing models are expensive to train and deploy, and it is also difficult to expand their knowledge beyond pre-training data without forgetting previous knowledge. This paper proposes a new neural network architecture, ModuleFormer, that leverages modularity to improve the efficiency and flexibility of large language models. ModuleFormer is based on the Sparse Mixture of Experts (SMoE). Unlike the previous SMoE-based modular language model, which requires domain-labeled data to learn domain-specific experts, ModuleFormer can induce modularity from uncurated data with its new load balancing and concentration losses. ModuleFormer is a modular architecture that includes two different types of modules: new stick-breaking attention heads and feedforward experts. Different modules are sparsely activated conditions on the input token during training and inference. In our experiment, we found that the modular architecture enables three important abilities for large pre-trained language models: 1) Efficiency, since ModuleFormer only activates a subset of its modules for each input token, thus it could achieve the same performance as dense LLMs with more than two times throughput; 2) Extendability, ModuleFormer is more immune to catastrophic forgetting than dense LLMs and can be easily extended with new modules to learn new knowledge that is not included in the training data; 3) Specialisation, finetuning ModuleFormer could specialize a subset of modules to the finetuning task and the task-unrelated modules could be easily pruned for a lightweight deployment.

翻译：大型语言模型（LLMs）已取得显著成果。然而，现有模型在训练和部署上成本高昂，且在不遗忘先前知识的情况下扩展预训练数据之外的知识十分困难。本文提出一种新的神经网络架构——ModuleFormer，利用模块化提升大型语言模型的效率与灵活性。ModuleFormer基于稀疏混合专家（SMoE）。与以往需要领域标注数据来学习领域特定专家的基于SMoE的模块化语言模型不同，ModuleFormer通过其新的负载均衡和集中损失函数，能从非整理数据中诱导出模块化特性。ModuleFormer是一种包含两类不同模块（新型断棒注意力头和前馈专家）的模块化架构。在训练和推理过程中，不同模块会根据输入令牌稀疏激活。实验中，我们发现模块化架构能为大型预训练语言模型带来三种重要能力：1）高效性：由于ModuleFormer对每个输入令牌仅激活部分模块，因此其吞吐量可达同等性能密集型LLM的两倍以上；2）可扩展性：ModuleFormer比密集型LLM更能抵抗灾难性遗忘，且可通过添加新模块轻松学习训练数据中未包含的新知识；3）专精性：微调ModuleFormer可使部分子模块专精于微调任务，而任务无关模块可被轻松剪枝以实现轻量化部署。