Foundation models have achieved great advances in multi-task learning with a unified interface of unimodal and multimodal tasks. However, the potential of such multi-task learners has not been exploited during transfer learning. In this work, we present a universal parameter-efficient transfer learning method, termed Predict-Interpolate Tuning ($\pi$-Tuning), for vision, language, and vision-language tasks. It aggregates the parameters of lightweight task-specific experts learned from similar tasks to aid the target downstream task. The task similarities are predicted in a unified modality-independent space, yielding a scalable graph to demonstrate task relationships. $\pi$-Tuning has several appealing benefits. First, it flexibly explores both intra- and inter-modal transferability between similar tasks to improve the accuracy and robustness of transfer learning, especially in data-scarce scenarios. Second, it offers a systematical solution for transfer learning with multi-task prediction-and-then-interpolation, compatible with diverse types of parameter-efficient experts, such as prompt and adapter. Third, an extensive study of task-level mutual benefits on 14 unimodal and 6 multimodal datasets shows that $\pi$-Tuning surpasses fine-tuning and other parameter-efficient transfer learning methods both in full-shot and low-shot regimes. The task graph also enables an in-depth interpretable analysis of task transferability across modalities.

翻译：基础模型通过统一接口处理单模态与多模态任务，已在多任务学习领域取得显著进展。然而，这类多任务学习器在迁移学习中的潜力尚未被充分挖掘。本文提出一种通用的参数高效迁移学习方法——预测-插值调优（$\pi$-Tuning），适用于视觉、语言及视觉-语言任务。该方法通过聚合从相似任务中学习到的轻量级任务特定专家参数来辅助目标下游任务，并在统一的模态无关空间中预测任务相似度，构建可扩展的任务关系图。$\pi$-Tuning具有以下优势：第一，灵活探索相似任务间的模态内与模态间可迁移性，在数据稀缺场景下尤其能提升迁移学习的准确性与鲁棒性；第二，提供基于"多任务预测-插值"的系统化迁移学习方案，兼容提示学习、适配器等各类参数高效专家模块；第三，在14个单模态和6个多模态数据集上的任务级互惠效应研究表明，无论是在全样本还是低样本场景下，$\pi$-Tuning均超越微调及其他参数高效迁移学习方法。任务关系图还实现了跨模态任务可迁移性的深度可解释分析。