We advance the field of Parameter-Efficient Fine-Tuning (PEFT) with our novel multi-adapter method, OrchMoE, which capitalizes on modular skill architecture for enhanced forward transfer in neural networks. Unlike prior models that depend on explicit task identification inputs, OrchMoE automatically discerns task categories, streamlining the learning process. This is achieved through an integrated mechanism comprising an Automatic Task Classification module and a Task-Skill Allocation module, which collectively deduce task-specific classifications and tailor skill allocation matrices. Our extensive evaluations on the 'Super Natural Instructions' dataset, featuring 1,600 diverse instructional tasks, indicate that OrchMoE substantially outperforms comparable multi-adapter baselines in terms of both performance and sample utilization efficiency, all while operating within the same parameter constraints. These findings suggest that OrchMoE offers a significant leap forward in multi-task learning efficiency.

翻译：我们通过提出新颖的多适配器方法OrchMoE，推动了参数高效微调（PEFT）领域的发展。该方法利用模块化技能架构，增强了神经网络的前向迁移能力。与依赖显式任务识别输入的先前模型不同，OrchMoE能够自动辨别任务类别，从而简化学习过程。这是通过一个包含自动任务分类模块和任务-技能分配模块的集成机制实现的，该机制共同推导出任务特定的分类并定制技能分配矩阵。我们在包含1,600个多样化指令任务的"Super Natural Instructions"数据集上进行的广泛评估表明，在相同参数约束下，OrchMoE在性能和样本利用效率上均显著优于同类多适配器基线模型。这些发现表明，OrchMoE在多任务学习效率方面实现了重大突破。