We propose Conditional Adapter (CoDA), a parameter-efficient transfer learning method that also improves inference efficiency. CoDA generalizes beyond standard adapter approaches to enable a new way of balancing speed and accuracy using conditional computation. Starting with an existing dense pretrained model, CoDA adds sparse activation together with a small number of new parameters and a light-weight training phase. Our experiments demonstrate that the CoDA approach provides an unexpectedly efficient way to transfer knowledge. Across a variety of language, vision, and speech tasks, CoDA achieves a 2x to 8x inference speed-up compared to the state-of-the-art Adapter approach with moderate to no accuracy loss and the same parameter efficiency.

翻译：我们提出条件适配器（CoDA），这是一种参数高效的迁移学习方法，同时提升了推理效率。CoDA在标准适配器方法的基础上进行了泛化，通过条件计算实现了一种平衡速度与准确性的新方式。该方法从现有稠密预训练模型出发，在新增少量参数和轻量训练阶段的基础上引入稀疏激活机制。实验表明，CoDA提供了一种非预期的知识迁移高效途径。在语言、视觉和语音等多项任务中，相较于当前最先进的适配器方法，CoDA在保持相同参数效率且几乎不损失准确率甚至无损失的情况下，实现了2至8倍的推理速度提升。