We present HyperLoader, a simple approach that combines different parameter-efficient fine-tuning methods in a multi-task setting. To achieve this goal, our model uses a hypernetwork to generate the weights of these modules based on the task, the transformer layer, and its position within this layer. Our method combines the benefits of multi-task learning by capturing the structure of all tasks while reducing the task interference problem by encapsulating the task-specific knowledge in the generated weights and the benefits of combining different parameter-efficient methods to outperform full-fine tuning. We provide empirical evidence that HyperLoader outperforms previous approaches in most datasets and obtains the best average performance across tasks in high-resource and low-resource scenarios.

翻译：本文提出HyperLoader，一种在多任务场景下整合不同参数高效微调方法的简洁方案。为实现该目标，本模型采用超网络架构，依据具体任务、Transformer层及其在层内的位置动态生成这些模块的权重。该方法既通过捕获所有任务的结构获得多任务学习的优势，又通过将任务特定知识封装于生成权重中以缓解任务干扰问题，同时结合多种参数高效方法的优点以超越全参数微调性能。我们通过实证研究表明，HyperLoader在多数数据集上优于现有方法，并在高资源与低资源场景下的跨任务平均性能评估中取得最优结果。