Foundation models have achieved great advances in multi-task learning with a unified interface of unimodal and multimodal tasks. However, the potential of such multi-task learners has not been exploited during transfer learning. In this work, we present a universal parameter-efficient transfer learning method, termed Predict-Interpolate Tuning ($\pi$-Tuning), for vision, language, and vision-language tasks. It aggregates the parameters of lightweight task-specific experts learned from similar tasks to aid the target downstream task. The task similarities are predicted in a unified modality-independent space, yielding a scalable graph to demonstrate task relationships. $\pi$-Tuning has several appealing benefits. First, it flexibly explores both intra- and inter-modal transferability between similar tasks to improve the accuracy and robustness of transfer learning, especially in data-scarce scenarios. Second, it offers a systematical solution for transfer learning with multi-task prediction-and-then-interpolation, compatible with diverse types of parameter-efficient experts, such as prompt and adapter. Third, an extensive study of task-level mutual benefits on 14 unimodal and 6 multimodal datasets shows that $\pi$-Tuning surpasses fine-tuning and other parameter-efficient transfer learning methods both in full-shot and low-shot regimes. The task graph also enables an in-depth interpretable analysis of task transferability across modalities.

翻译：基础模型通过统一接口实现单模态与多模态任务的多任务学习，已取得显著进展。然而，这类多任务学习器在迁移学习中的潜力尚未被充分挖掘。本文提出一种通用的参数高效迁移学习方法——预测-插值调优（$\pi$-Tuning），适用于视觉、语言及视觉-语言任务。该方法通过聚合从相似任务中学习到的轻量级任务特定专家参数，辅助目标下游任务。任务相似性在统一的模态无关空间中进行预测，生成可扩展图以展示任务关联。$\pi$-Tuning具有多项优势：第一，灵活探索相似任务间的模态内与跨模态迁移性，提升迁移学习的准确性与鲁棒性，尤其在数据稀缺场景下表现突出；第二，提供基于"多任务预测-插值"的系统性迁移学习解决方案，兼容提示学习、适配器等多种参数高效专家模块；第三，在14个单模态与6个多模态数据集上的任务级互惠效益全面研究表明，$\pi$-Tuning在全样本与小样本场景下均优于微调及其他参数高效迁移学习方法。任务图还可实现跨模态任务迁移性的深度可解释分析。