Modular and composable transfer learning is an emerging direction in the field of Parameter Efficient Fine-Tuning, as it enables neural networks to better organize various aspects of knowledge, leading to improved cross-task generalization. In this paper, we introduce a novel approach Customized Polytropon C-Poly that combines task-common skills and task-specific skills, while the skill parameters being highly parameterized using low-rank techniques. Each task is associated with a customizable number of exclusive specialized skills and also benefits from skills shared with peer tasks. A skill assignment matrix is jointly learned. To evaluate our approach, we conducted extensive experiments on the Super-NaturalInstructions and the SuperGLUE benchmarks. Our findings demonstrate that C-Poly outperforms fully-shared, task-specific, and skill-indistinguishable baselines, significantly enhancing the sample efficiency in multi-task learning scenarios.

翻译：模块化与可组合的迁移学习是参数高效微调领域的一个新兴方向，它能使神经网络更好地组织各类知识，从而提升跨任务泛化能力。本文提出了一种新颖方法——定制化Polytropon（C-Poly），该方法结合了任务通用技能与任务特定技能，并通过低秩技术对技能参数进行高度参数化表示。每个任务关联可定制数量的专属专门技能，同时也能从与其他任务共享的技能中受益。我们联合学习一个技能分配矩阵。为评估该方法，我们在Super-NaturalInstructions和SuperGLUE基准测试上进行了广泛实验。结果表明，C-Poly在性能上优于全共享、任务特定以及技能无区别的基线方法，在多任务学习场景中显著提升了样本效率。