Recently, there has been a widespread proliferation of "expert" language models that are specialized to a specific task or domain through parameter-efficient fine-tuning. How can we recycle large collections of expert language models to improve zero-shot generalization to unseen tasks? In this work, we propose Post-Hoc Adaptive Tokenwise Gating Over an Ocean of Specialized Experts (PHATGOOSE), which learns to route among specialized modules that were produced through parameter-efficient fine-tuning. Unlike past methods that learn to route among specialized models, PHATGOOSE explores the possibility that zero-shot generalization will be improved if different experts can be adaptively chosen for each token and at each layer in the model. Crucially, our method is post-hoc - it does not require simultaneous access to the datasets used to create the specialized models and only requires a modest amount of additional compute after each expert model is trained. In experiments covering a range of specialized model collections and zero-shot generalization benchmarks, we find that PHATGOOSE outperforms past methods for post-hoc routing and, in some cases, outperforms explicit multitask training (which requires simultaneous data access). To better understand the routing strategy learned by PHATGOOSE, we perform qualitative experiments to validate that PHATGOOSE's performance stems from its ability to make adaptive per-token and per-module expert choices. We release all of our code to support future work on improving zero-shot generalization by recycling specialized experts.

翻译：近期，通过参数高效微调针对特定任务或领域进行优化的“专家”语言模型广泛涌现。我们如何利用大量专家语言模型的集合来提升对未见任务的零样本泛化能力？本文提出了一种名为“面向专业专家海洋的后验自适应逐词元门控机制”（PHATGOOSE），该机制学习在通过参数高效微调产生的专业模块之间进行路由。与先前需要学习在专业模型间路由的方法不同，PHATGOOSE探索了一种可能性：若能在模型的每一层为每个词元自适应选择不同专家，零样本泛化能力将得到提升。关键的是，我们的方法是后验式的——它无需同时访问用于创建专业模型的数据集，且仅需在每个专家模型训练完成后投入适度额外的计算资源。在涵盖多种专业模型集合与零样本泛化基准的实验评估中，PHATGOOSE在性能上超越了以往的后验路由方法，且在某些情况下甚至优于需要同步数据访问的显式多任务训练。为深入理解PHATGOOSE习得的路由策略，我们开展了定性实验，验证了其性能优势源于对每个词元与每个模块进行自适应专家选择的能力。我们已公开全部代码，以支持未来通过复用专业专家提升零样本泛化的研究工作。