Mixture-of-experts (MoE) architecture has been proven a powerful method for diverse tasks in training deep models in many applications. However, current MoE implementations are task agnostic, treating all tokens from different tasks in the same manner. In this work, we instead design a novel method that incorporates task information into MoE models at different granular levels with shared dynamic task-based adapters. Our experiments and analysis show the advantages of our approaches over the dense and canonical MoE models on multi-task multilingual machine translations. With task-specific adapters, our models can additionally generalize to new tasks efficiently.

翻译：混合专家（MoE）架构已被证明是训练深度模型时处理多种任务的有效方法，广泛应用于诸多领域。然而，当前的MoE实现是任务无关的，即对来自不同任务的所有标记（token）采用相同处理方式。本研究设计了一种新颖的方法，通过共享动态基于任务的适配器（adapter），在多个粒度层次上将任务信息融入MoE模型。实验与分析表明，与密集模型和标准MoE模型相比，我们的方法在多任务多语言机器翻译中具有优势。借助任务特定的适配器，我们的模型还能高效泛化至新任务。