Large Language Models (LLMs) have demonstrated considerable proficiency in general natural language processing (NLP) tasks. Instruction tuning, a successful paradigm, enhances the ability of LLMs to follow natural language instructions and exhibit robust generalization across a wide range of tasks. However, these models often encounter performance limitations across multiple tasks due to constrained model capacity. Expanding this capacity during the instruction tuning phase poses significant challenges. To address this issue, we introduce a novel approach, Parameter-Efficient Sparsity Crafting (PESC), which transitions dense models to sparse models using a Mixture of Experts (MoE) architecture. PESC integrates adapters into the MoE layers of sparse models, differentiating experts without altering the individual weights within these layers. This method significantly reduces computational costs and GPU memory requirements, facilitating model capacity expansion through a minimal increase in parameters via the inserted adapters. Our empirical evaluation demonstrates the effectiveness of the PESC method. Using PESC during instruction tuning, our sparse models, dubbed Camelidae outperform all other opensource sparse models and exhibit superior general capabilities compared to GPT3.5.

翻译：大型语言模型在通用自然语言处理任务中展现出显著能力。指令微调作为一种成功范式，增强了语言模型遵循自然语言指令并在广泛任务中展现稳健泛化能力。然而，受限于模型容量，这些模型在多任务处理中常遇到性能瓶颈。在指令微调阶段扩展模型容量面临重大挑战。针对这一问题，我们提出一种新型方法——参数高效稀疏构建（PESC），通过专家混合（MoE）架构将密集模型过渡为稀疏模型。PESC将适配器集成到稀疏模型的MoE层中，在不改变层内各权重的前提下实现专家差异化。该方法显著降低计算成本与GPU内存需求，通过插入适配器仅增加少量参数即可实现模型容量扩展。实证评估验证了PESC方法的有效性。在指令微调中应用PESC后，我们命名为Camelidae的稀疏模型性能超越所有其他开源稀疏模型，并展现出优于GPT3.5的通用能力。